- Replace per‑chunk sorting with aggregation of total size per section.
- Compute total characters and chunk count for each document.
- Sort sections by total size and display the top five, including part count when applicable.
- Renamed `partIndex`/`totalParts` to `partindex`/`totalparts` in `section_retriever.ts`, `spec_retriever.ts`, and document creation logic.
- Adjusted sorting logic to use the new `partindex` field.
- Updated metadata handling and string interpolation to reference the lowercase keys.
- Fixed import path for `HTMLTextSplitter` and moved the file to `setup/textsplitters/HtmlTextSplitter.ts`.
- Added HTML preprocessing utilities (`addNewlinesAfterBlocks`, `convertTablesToMarkdown`) in ingestion workflow.
- Integrated table-to‑markdown conversion and newline insertion to improve text splitting and document structure.
- Verify that `splitDocuments` correctly prepends the provided `chunkHeader` to each chunk.
- Ensure `splitDocuments` returns an empty array when the source HTML yields no chunks.
Introduce a new manual test file `test/manual/test-spec-retriever.ts` that demonstrates how to use the `spec_retriever` agent tool with a query. The script loads embeddings, connects to the LanceDB storage, creates the tool, executes it, and prints the result. This provides a runnable example for developers.
bendtherules