add ChunkInfo interface, improve chunk handling typings, and print ingest summary statistics. This introduces typed chunk metadata, refactors related code, and adds a summary report for document sizes, sections, and chunk distribution
- Add .vscode/extensions.json recommending oven.bun-vscode.
- Add .vscode/launch.json with Bun debug configurations for various scripts.
- Add .vscode/settings.json for TypeScript SDK, debug options, and file exclusions.
- Remove .vscode/ entry from .gitignore.
- Refactor ingest.ts: increase chunk size to 8192 and overlap to 200, raise large document threshold, remove <h1>/<h2> separators, skip sections containing only headings or minimal content, and add warnings for very small chunks.
- Remove construction of `partRef` and parent section annotations from chunk text
- Directly extract and trim text from HTML chunk using cheerio
- Streamline document creation logic without additional references.
- Added EMBEDDING_MODEL and RERANKER_MODEL to `constants.ts`.
- Updated imports to use these constants in `agent.ts`, `agent_tools/reranker.ts`, and `setup/ingest.ts`.
- Replaced hard‑coded model strings with the new constants for Ollama embeddings and reranker.
add HTML‑aware text splitter, raise batch size and large‑doc threshold, enhance progress spinner and error logging, rename ingestSpec to buildSpecDocuments and introduce hierarchical section processing.
- Move full‑section Document creation into an `else` block so it only occurs when no sub‑documents were created.
- Remove premature return that skipped adding the full section.
- Add post‑processing loop to log a warning for any final document exceeding `LARGE_DOC_THRESHOLD`.
bendtherules