# Plan for building a RAG-like pipeline for language specification exploration

**Technology Stack:**
- **Framework:** LlamaIndex.js
- **Graph Storage:** LlamaIndex `SimpleGraphStore` (file-based)
- **Embeddings:** Local Ollama

## High-Level Architecture

The system will have four main components:

1.  **Data Ingestion:** A pipeline to process the HTML specification and the Javascript code.
2.  **Data Storage:** A combination of a vector store and a graph store, both managed by LlamaIndex.
3.  **RAG Pipeline:** A retrieval-augmented generation pipeline built with LlamaIndex.
4.  **Agentic Chat Interface:** An interactive chat interface for the user to interact with the system.

## Phase 1: Data Ingestion and Basic RAG

This phase focuses on getting the basic RAG pipeline working with LlamaIndex.

*   **Task 1: Setup Project and Dependencies:**
    *   Initialize a Node.js project.
    *   Install `llamaindex`.
    *   Set up Ollama for local embeddings.
    *   Clone the `engine262` repository from `https://github.com/engine262/engine262` into the `./engine262` folder.

*   **Task 2: HTML Specification Parsing:**
    *   Write a script to recursively read all HTML files from the `./spec-built/multipage` folder.
    *   Use a library like `cheerio` to parse the HTML and extract the main text content.
    *   Use LlamaIndex's `SimpleNodeParser` to break down the content into meaningful chunks (nodes).

*   **Task 3: Javascript Code Parsing:**
    *   Write a script to read the Javascript files from the `./engine262/src` folder.
    *   Initially, treat the code as plain text and parse it into nodes.

*   **Task 4: Vector Database Integration with LlamaIndex:**
    *   Configure LlamaIndex to use your local Ollama instance for generating embeddings by using the `OllamaEmbedding` class.
    *   LlamaIndex will manage the documents and embeddings in a vector store. You can start with an in-memory store.

*   **Task 5: Basic RAG Pipeline:**
    *   Use LlamaIndex's `VectorStoreIndex` to build an index over your parsed documents.
    *   Create a `QueryEngine` from the index to ask questions.

*   **Task 6: Simple CLI Interface:**
    *   Create a simple command-line interface to ask questions and see the results from the LlamaIndex query engine.

## Phase 2: Graph Integration with Graphology

This phase will enhance the retrieval process by modeling the structure of the specification and the code using Graphology. LlamaIndex.js's native graph store is currently limited, so we'll use a custom mapping.

*   **Task 1: Graph Storage Setup:**
    *   Use `graphology` to build an in-memory graph.
    *   Persist the graph to a `graph.json` file for persistence.

*   **Task 2: Enhance Parsers to Extract Relationships:**
    *   **HTML Parser:** When parsing the spec, extract section titles (e.g., "If Statement") and their IDs.
    *   **JS Parser:** Use `acorn` or `@babel/parser` to create an AST of the Javascript code and extract function names. Create a mapping from function names to specification section titles (e.g., `Evaluate_IfStatement` -> `If Statement`).

*   **Task 3: Populate Graph Store:**
    *   Write a script to build the graph:
        *   Create nodes for specification sections (`SpecSection`) and Javascript functions (`JSFunction`).
        *   Create relationships for links between spec sections (`LINKS_TO`) and for functions implementing a spec section (`IMPLEMENTS`).
    *   Save this graph to `graph.json`.

*   **Task 4: Enhance Retriever with Custom Graph Lookup:**
    *   Create a custom retriever that first looks up the relevant code/spec in the graph and then uses the vector store for detailed retrieval.

## Phase 3: Agentic Chat and Tool Use

This phase focuses on building the interactive and "smart" agent using LlamaIndex's capabilities.

*   **Task 1: Agent Framework:**
    *   Use LlamaIndex's agent classes, like `ChatEngine` or `QueryEngine` with tools, to create the agent.
    *   Configure the agent to use your OpenAI-compatible service for generation.

*   **Task 2: Define Tools:**
    *   LlamaIndex allows you to define tools. The primary tool will be a `QueryEngineTool` that uses the combined vector and graph index from the previous phases.
    *   You could also create more specific tools, like one to directly query the graph for structural information.

*   **Task 3: Build the Chat Interface:**
    *   Create a web-based chat interface (e.g., using React or Vue).
    *   This interface will interact with the LlamaIndex agent endpoint.

