add HTML‑aware text splitter, raise batch size and large‑doc threshold, enhance progress spinner and error logging, rename ingestSpec to buildSpecDocuments and introduce hierarchical section processing.
- Consolidate all structural element breakdowns into a single `breakDownSection` function using an `alwaysBreak` flag.
- Introduce `BREAKDOWN_TAGS` configuration array containing metadata (tag, alwaysBreak, title/id selectors) for each element type.
- Replace previous multi‑phase approach with a unified sequential breakdown flow: `emu-clause` (always extracts children) → `emu-table` → `emu-grammar` → `td` → `p`.
- Add inline markers `[Subsection available: title "X" at sectionid: ID]` where content is removed, enabling parent awareness.
- Update documentation to reflect the new unified breakdown logic, tag table, and hierarchical ID format.
- Adjust threshold handling, recursion depth, and metadata tracking to work with the new unified approach.
Added a detailed markdown plan outlining fixes for parent section awareness, sequential tag‑based breakdown, and recursive processing of large sections. Includes implementation details, configuration constants, and updated ingestion logic. This new plan file is located at `.opencode/plans/1774872666261-parent-breakdown-fix.md`.
- Move full‑section Document creation into an `else` block so it only occurs when no sub‑documents were created.
- Remove premature return that skipped adding the full section.
- Add post‑processing loop to log a warning for any final document exceeding `LARGE_DOC_THRESHOLD`.
Added a comprehensive markdown plan (.opencode/plans/1774872124705-glowing-river.md) outlining the migration from LlamaIndex to LangChain.js with LanceDB. Includes updated dependencies, file modification list, and detailed code changes for `ingest.ts` and `agent.ts`. This serves as documentation for the migration process.
Search for `emu-clause` elements now performed within the `#spec-container` element instead of the entire document, preventing accidental matches outside the spec container.
- Log the number of specification HTML files found in `SPEC_DIR`.
- Emit a warning if no specification HTML files are present.
- Log the number of code files found in `CODE_DIR`.
- Emit a warning if no code files are present.
Update init_engine262.sh to add the engine262 repository as a squashed subtree instead of a shallow submodule. Adjust usage comments, add existence check for the target directory, and update echo messages accordingly.
Introduces `setup/init_engine262.sh`, a Bash utility that clones the `engine262` repository as a shallow submodule (depth 1) into `./engine262`. The script adds the submodule, initializes it, and provides a usage comment. It ensures robust execution with `set -euo pipefail`.
- Added `constants.mjs` defining `STORAGE_DIR` and `GRAPH_FILE`.
- Replaced hard‑coded path strings in `agent.mjs`, `build_graph.mjs`, and `ingest.mjs` with imports from the new constants module.
- Updated console log in `ingest.mjs` to reference `STORAGE_DIR`.
- Adjusted `GRAPH_FILE` path to `./graphology/graph.json` via the constant.
- Include common directories (node_modules, dist, build)
- Ignore logs, OS files, coverage, IDE settings, and environment files
- Add placeholders for package lock files and generated docs.
bendtherules