mirror of
https://github.com/bendtherules/ask262.git
synced 2026-08-18 13:21:55 +00:00
feat: Impl HtmlTextSplitter
This commit is contained in:
@@ -0,0 +1,556 @@
|
||||
---
|
||||
model: openai/gpt-5.4
|
||||
---
|
||||
|
||||
# HTMLTextSplitter Implementation Plan
|
||||
|
||||
## Overview
|
||||
|
||||
Build a new `HTMLTextSplitter` that extends LangChain's `TextSplitter` and splits HTML by traversing the parsed DOM tree instead of slicing raw HTML strings.
|
||||
|
||||
This version is intentionally narrow:
|
||||
- size metric uses cheerio text extraction only
|
||||
- separators are CSS selectors
|
||||
- `neverBreakWithin` is respected unless `maxChunkSize` forces a deeper split
|
||||
- no `chunkOverlap` support
|
||||
- output is text chunks, not reconstructed partial HTML
|
||||
|
||||
## Confirmed Decisions
|
||||
|
||||
1. Use a traversal-based algorithm, not source offsets.
|
||||
2. Only split when needed for size control.
|
||||
3. Separator matches are soft structural hints, not unconditional split points.
|
||||
4. If a separator subtree is too large, recurse into that subtree.
|
||||
5. `neverBreakWithin` wins unless a node exceeds `maxChunkSize`; then recurse as a last resort.
|
||||
6. Ignore separator matches inside protected subtrees during normal traversal.
|
||||
7. V1 emits text chunks, not valid partial HTML.
|
||||
8. Traverse both element nodes and text nodes.
|
||||
9. Normalize output after boundary-aware joining so block-ish tags create readable spacing without adding synthetic spaces between inline tags.
|
||||
10. When a direct child matches a separator during decomposition, it stays with the following content.
|
||||
11. `partIndex` and `totalParts` are tracked per input document.
|
||||
12. `chunkOverlap` is not supported.
|
||||
13. Scope is the splitter itself, with ingest integration called out as a follow-up.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Extend `TextSplitter` from `@langchain/textsplitters`.
|
||||
2. Accept HTML input and parse with `cheerio`.
|
||||
3. Use CSS selectors for `separators` and `neverBreakWithin`.
|
||||
4. Use `chunkSize` as a soft target and `maxChunkSize` as a hard limit.
|
||||
5. Produce chunks whose content is text derived from the traversed HTML.
|
||||
6. Recurse only when needed; avoid unnecessary fragmentation.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
1. Preserving exact HTML markup in chunk output.
|
||||
2. Supporting `chunkOverlap`.
|
||||
3. Building a full `setup/ingest.ts` migration in this plan.
|
||||
4. Reproducing browser-accurate `innerText` behavior.
|
||||
|
||||
## Public Interface
|
||||
|
||||
```typescript
|
||||
import { type Document } from "@langchain/core/documents";
|
||||
import { TextSplitter, type TextSplitterParams } from "@langchain/textsplitters";
|
||||
|
||||
export interface HTMLTextSplitterParams extends TextSplitterParams {
|
||||
/**
|
||||
* CSS selectors that act as preferred structural split points.
|
||||
* They are only used when a subtree needs to be broken down.
|
||||
*/
|
||||
separators?: string[];
|
||||
|
||||
/**
|
||||
* CSS selectors for nodes that should stay atomic unless maxChunkSize forces recursion.
|
||||
*/
|
||||
neverBreakWithin?: string[];
|
||||
|
||||
/**
|
||||
* Absolute hard limit for chunk text length.
|
||||
* Defaults to Infinity if omitted.
|
||||
*/
|
||||
maxChunkSize?: number;
|
||||
}
|
||||
|
||||
export class HTMLTextSplitter extends TextSplitter {
|
||||
constructor(fields?: Partial<HTMLTextSplitterParams>);
|
||||
|
||||
/**
|
||||
* Splits HTML input into normalized text chunks.
|
||||
*
|
||||
* Important: this splitter does not preserve HTML markup in output.
|
||||
* It uses parsed HTML structure for traversal and sizing, but emits text only.
|
||||
*/
|
||||
splitText(text: string): Promise<string[]>;
|
||||
|
||||
/**
|
||||
* Splits HTML documents into text-only child documents.
|
||||
*
|
||||
* Important: child document `pageContent` contains normalized text, not HTML.
|
||||
* Parent metadata is preserved and per-document part metadata is added.
|
||||
*/
|
||||
splitDocuments(documents: Document[]): Promise<Document[]>;
|
||||
}
|
||||
```
|
||||
|
||||
## Output Semantics
|
||||
|
||||
1. `splitText()` accepts HTML and returns `string[]` text chunks.
|
||||
2. Each chunk is text content extracted from a DOM subtree or subtree fragment.
|
||||
3. `splitDocuments()` preserves parent metadata and adds per-document `partIndex` and `totalParts`.
|
||||
4. Final chunk text is normalized after DOM-aware joining; block-ish tag boundaries may add spacing, while inline tag boundaries do not add synthetic spaces.
|
||||
|
||||
## Size Semantics
|
||||
|
||||
1. `chunkSize` is the target size.
|
||||
2. `maxChunkSize` is the hard cap when provided.
|
||||
3. The size function is based on the same normalized text content used for output segments.
|
||||
4. Segment text is derived from cheerio text extraction, joined with DOM-aware boundaries, then whitespace-normalized before measurement.
|
||||
5. No separate text-length implementation should be maintained outside that semantic.
|
||||
6. If `maxChunkSize` is omitted, treat it as `Infinity`.
|
||||
|
||||
## Core Algorithm
|
||||
|
||||
### Mental Model
|
||||
|
||||
Treat each DOM node as a unit that can either:
|
||||
- fit as-is
|
||||
- stay atomic because it is protected and still within `maxChunkSize`
|
||||
- be recursively decomposed into children because it is too large
|
||||
|
||||
Separators only matter when a node must be broken down. They influence recursion order; they do not force a split on their own.
|
||||
|
||||
### Traversal Strategy
|
||||
|
||||
1. Parse the HTML with cheerio.
|
||||
2. Select a traversal root:
|
||||
- use `body` children if a body exists
|
||||
- otherwise use root children
|
||||
3. Walk top-level nodes in document order.
|
||||
4. Convert each node into one or more text segments using recursive decomposition.
|
||||
5. Merge resulting segments into final chunks while respecting `chunkSize` and `maxChunkSize`.
|
||||
|
||||
### Recursive Decomposition Rules
|
||||
|
||||
For a node `N`:
|
||||
|
||||
1. Compute `nodeText` using the shared `getNodeText()` helper so both element nodes and text nodes follow the same trimming semantics.
|
||||
2. If `nodeText` is empty, skip it.
|
||||
3. If `nodeText.length <= chunkSize`, emit it as one segment.
|
||||
4. If `N` matches `neverBreakWithin` and `nodeText.length <= maxChunkSize`, emit it as one segment.
|
||||
5. Otherwise, recurse into children.
|
||||
6. If `N` has no element/text children that can further subdivide the content, force-split the node text with `RecursiveCharacterTextSplitter`.
|
||||
|
||||
### Separator-Aware Recursion
|
||||
|
||||
When recursing into a node with child elements:
|
||||
|
||||
1. Partition children into traversal groups.
|
||||
2. Prefer group boundaries at direct child elements matching `separators`.
|
||||
3. A separator child starts a new group and stays with the following sibling content rather than the preceding content.
|
||||
4. If no separator child exists, recurse in plain DOM order.
|
||||
5. Ignore separator matches inside any subtree currently treated as protected.
|
||||
|
||||
## Internal Types
|
||||
|
||||
```typescript
|
||||
interface Segment {
|
||||
text: string;
|
||||
isProtected: boolean;
|
||||
sourceKind: "node" | "forced-split";
|
||||
}
|
||||
```
|
||||
|
||||
V1 should keep internal types local to the implementation file unless reuse appears during build.
|
||||
|
||||
## Detailed Behavior
|
||||
|
||||
### 1. Protected Nodes
|
||||
|
||||
If a node matches `neverBreakWithin`, keep it atomic while its text length is `<= maxChunkSize`. Recurse into it only if it exceeds `maxChunkSize`.
|
||||
|
||||
If `maxChunkSize` is `Infinity`, protected nodes are never forced open by hard-cap rules.
|
||||
|
||||
### 2. Forced Splitting
|
||||
|
||||
When recursion reaches a node whose text still exceeds `maxChunkSize`, and there is no useful child structure left, use `RecursiveCharacterTextSplitter` on the extracted text.
|
||||
|
||||
Use it with:
|
||||
- `chunkSize: this.maxChunkSize`
|
||||
- `chunkOverlap: 0`
|
||||
|
||||
This is the final fallback that guarantees the hard limit.
|
||||
|
||||
If `maxChunkSize` is `Infinity`, this fallback is not used for hard-cap enforcement.
|
||||
|
||||
### 3. Merging
|
||||
|
||||
After recursion produces a flat list of text segments:
|
||||
|
||||
1. Start a new chunk.
|
||||
2. Append the next segment if the combined length stays within `chunkSize`.
|
||||
3. If appending would exceed `chunkSize`, flush the current chunk and start a new one.
|
||||
4. If any individual segment is already larger than `chunkSize` but not larger than `maxChunkSize`, allow that segment to become its own chunk.
|
||||
5. Join merged segment text using DOM-aware boundaries and whitespace normalization.
|
||||
6. No overlap is applied.
|
||||
|
||||
When `maxChunkSize` is `Infinity`, a single segment may grow beyond `chunkSize` if it cannot be decomposed further without violating other rules.
|
||||
|
||||
## Pseudocode
|
||||
|
||||
```typescript
|
||||
async splitText(html: string): Promise<string[]> {
|
||||
const $ = cheerio.load(html);
|
||||
const roots = $("body").length > 0 ? $("body").contents().toArray() : $.root().contents().toArray();
|
||||
|
||||
const segments: Segment[] = [];
|
||||
for (const node of roots) {
|
||||
segments.push(...(await this.collectSegments($, node, false)));
|
||||
}
|
||||
|
||||
return this.mergeSegments(segments);
|
||||
}
|
||||
|
||||
private async collectSegments(
|
||||
$: cheerio.CheerioAPI,
|
||||
node: cheerio.AnyNode,
|
||||
inProtectedAncestor: boolean,
|
||||
): Promise<Segment[]> {
|
||||
const text = this.getNodeText($, node);
|
||||
if (!text) return [];
|
||||
|
||||
const isProtected = !inProtectedAncestor && this.matchesAny($, node, this.neverBreakWithin);
|
||||
|
||||
if (text.length <= this.chunkSize) {
|
||||
return [{ text, isProtected, sourceKind: "node" }];
|
||||
}
|
||||
|
||||
if (isProtected && text.length <= this.maxChunkSize) {
|
||||
return [{ text, isProtected, sourceKind: "node" }];
|
||||
}
|
||||
|
||||
const childNodes = this.getChildNodesForRecursion(node);
|
||||
if (childNodes.length === 0) {
|
||||
return this.forceSplitText(text);
|
||||
}
|
||||
|
||||
const childGroups = this.groupChildrenForDecomposition($, childNodes, isProtected || inProtectedAncestor);
|
||||
const segments: Segment[] = [];
|
||||
for (const group of childGroups) {
|
||||
for (const child of group) {
|
||||
segments.push(...(await this.collectSegments($, child, isProtected || inProtectedAncestor)));
|
||||
}
|
||||
}
|
||||
|
||||
return segments.length > 0 ? segments : this.forceSplitText(text);
|
||||
}
|
||||
```
|
||||
|
||||
## Utils Pseudocode
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Returns true when a tag node matches any configured CSS selector.
|
||||
*
|
||||
* Needed so separator and protected-node checks share the same selector logic
|
||||
* and non-tag nodes are ignored safely.
|
||||
*/
|
||||
private matchesAny(
|
||||
$: cheerio.CheerioAPI,
|
||||
node: cheerio.AnyNode,
|
||||
selectors: string[],
|
||||
): boolean {
|
||||
if (node.type !== "tag") {
|
||||
return false;
|
||||
}
|
||||
|
||||
for (const selector of selectors) {
|
||||
if ($(node).is(selector)) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts the canonical text value for a node.
|
||||
*
|
||||
* Needed so sizing and emitted output use the same trimmed text semantics for
|
||||
* both element nodes and raw text nodes.
|
||||
*/
|
||||
private getNodeText($: cheerio.CheerioAPI, node: cheerio.AnyNode): string {
|
||||
if (node.type === "text") {
|
||||
return (node.data ?? "").trim();
|
||||
}
|
||||
|
||||
return $(node).text().trim();
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the child nodes that are meaningful recursion units.
|
||||
*
|
||||
* Needed to recurse through DOM structure without carrying empty whitespace-only
|
||||
* text nodes or unsupported node types through the algorithm.
|
||||
*/
|
||||
private getChildNodesForRecursion(node: cheerio.AnyNode): cheerio.AnyNode[] {
|
||||
if (!("children" in node) || !Array.isArray(node.children)) {
|
||||
return [];
|
||||
}
|
||||
|
||||
return node.children.filter((child) => {
|
||||
if (child.type === "text") {
|
||||
return (child.data ?? "").trim().length > 0;
|
||||
}
|
||||
|
||||
return child.type === "tag";
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Groups child nodes so separator nodes start a new group.
|
||||
*
|
||||
* Needed because separators represent preferred section boundaries. Grouping
|
||||
* makes "separator plus following content" explicit during recursion.
|
||||
*/
|
||||
private groupChildrenForDecomposition(
|
||||
$: cheerio.CheerioAPI,
|
||||
childNodes: cheerio.AnyNode[],
|
||||
inProtectedTree: boolean,
|
||||
): cheerio.AnyNode[][] {
|
||||
if (inProtectedTree || this.separators.length === 0) {
|
||||
return [childNodes];
|
||||
}
|
||||
|
||||
const groups: cheerio.AnyNode[][] = [];
|
||||
let currentGroup: cheerio.AnyNode[] = [];
|
||||
|
||||
const flush = () => {
|
||||
if (currentGroup.length > 0) {
|
||||
groups.push(currentGroup);
|
||||
currentGroup = [];
|
||||
}
|
||||
};
|
||||
|
||||
for (const child of childNodes) {
|
||||
if (this.matchesAny($, child, this.separators)) {
|
||||
flush();
|
||||
}
|
||||
|
||||
currentGroup.push(child);
|
||||
}
|
||||
|
||||
flush();
|
||||
return groups;
|
||||
}
|
||||
|
||||
/**
|
||||
* Applies the final plain-text fallback when a node cannot be decomposed further.
|
||||
*
|
||||
* Needed to enforce a finite hard cap with `RecursiveCharacterTextSplitter`
|
||||
* once DOM structure is exhausted.
|
||||
*/
|
||||
private async forceSplitText(text: string): Promise<Segment[]> {
|
||||
if (!Number.isFinite(this.maxChunkSize)) {
|
||||
return [{ text: text.trim(), isProtected: false, sourceKind: "forced-split" }];
|
||||
}
|
||||
|
||||
const fallback = new RecursiveCharacterTextSplitter({
|
||||
chunkSize: this.maxChunkSize,
|
||||
chunkOverlap: 0,
|
||||
});
|
||||
|
||||
const chunks = await fallback.splitText(text);
|
||||
return chunks
|
||||
.map((chunk) => chunk.trim())
|
||||
.filter(Boolean)
|
||||
.map((chunk) => ({
|
||||
text: chunk,
|
||||
isProtected: false,
|
||||
sourceKind: "forced-split" as const,
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* Merges flat segments into final chunks using `chunkSize` as a soft target.
|
||||
*
|
||||
* Needed to keep chunk assembly separate from DOM traversal and to normalize the
|
||||
* final text output by joining segment text with single spaces.
|
||||
*/
|
||||
private mergeSegments(segments: Segment[]): string[] {
|
||||
const chunks: string[] = [];
|
||||
let currentParts: string[] = [];
|
||||
let currentLength = 0;
|
||||
|
||||
const flush = () => {
|
||||
if (currentParts.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
chunks.push(currentParts.join(" "));
|
||||
currentParts = [];
|
||||
currentLength = 0;
|
||||
};
|
||||
|
||||
for (const segment of segments) {
|
||||
const nextLength = currentLength === 0
|
||||
? segment.text.length
|
||||
: currentLength + 1 + segment.text.length;
|
||||
|
||||
if (currentParts.length > 0 && nextLength > this.chunkSize) {
|
||||
flush();
|
||||
}
|
||||
|
||||
currentParts.push(segment.text);
|
||||
currentLength = currentLength === 0
|
||||
? segment.text.length
|
||||
: currentLength + 1 + segment.text.length;
|
||||
}
|
||||
|
||||
flush();
|
||||
return chunks;
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Class Skeleton
|
||||
|
||||
1. Create `textsplitters/html_text_splitter.ts`.
|
||||
2. Define `HTMLTextSplitterParams`.
|
||||
3. Extend `TextSplitter`.
|
||||
4. Reject unsupported overlap in the constructor:
|
||||
- if `chunkOverlap !== 0`, throw a clear error
|
||||
5. Default `maxChunkSize` to `Infinity` when omitted.
|
||||
|
||||
### Phase 2: Tree Traversal Helpers
|
||||
|
||||
1. `matchesAny()` for selector matching.
|
||||
2. `getNodeText()` using cheerio text extraction and trimming semantics shared with output.
|
||||
3. `getChildNodesForRecursion()` to expose both text and element children in DOM order.
|
||||
4. `groupChildrenForDecomposition()` to build explicit child groups, with separator nodes grouped with following content.
|
||||
|
||||
### Phase 3: Recursive Collection
|
||||
|
||||
1. Implement `collectSegments()`.
|
||||
2. Preserve document order.
|
||||
3. Skip empty/whitespace-only results.
|
||||
|
||||
### Phase 4: Final Fallback
|
||||
|
||||
1. Implement `forceSplitText()` with `RecursiveCharacterTextSplitter`.
|
||||
2. Use `maxChunkSize` as the chunk size when it is finite.
|
||||
3. Use `chunkOverlap: 0`.
|
||||
4. Trim and drop empty results.
|
||||
|
||||
### Phase 5: Merge And Document Support
|
||||
|
||||
1. Merge segments into final chunks using `chunkSize` as the soft target.
|
||||
2. Implement `splitDocuments()` by mapping each input document through `splitText()`.
|
||||
3. Preserve original metadata and add per-document `partIndex` / `totalParts`.
|
||||
|
||||
## Edge Cases
|
||||
|
||||
### Empty HTML
|
||||
|
||||
Return `[]`.
|
||||
|
||||
### Plain Text Input
|
||||
|
||||
Cheerio will still parse it; return one or more text chunks based on size.
|
||||
|
||||
### Deeply Nested Protected Content
|
||||
|
||||
Keep it atomic unless it exceeds `maxChunkSize`; then recurse into its children and only force-split as a last resort.
|
||||
|
||||
If `maxChunkSize` is `Infinity`, it remains atomic unless some ancestor decomposition requires visiting its children.
|
||||
|
||||
### Separator Inside Protected Content
|
||||
|
||||
Ignore the separator during normal traversal.
|
||||
|
||||
### Large Text Node With No Useful Child Structure
|
||||
|
||||
Use text fallback splitting.
|
||||
|
||||
### Consecutive Small Nodes
|
||||
|
||||
Merge them up to `chunkSize`.
|
||||
|
||||
### A Single Segment Between chunkSize And maxChunkSize
|
||||
|
||||
Allow it as a standalone chunk.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Use Bun's built-in test runner and add automated tests under `test/textsplitters/`.
|
||||
|
||||
### Minimum Verification
|
||||
|
||||
1. `bunx tsc --noEmit`
|
||||
2. automated splitter tests under `test/textsplitters/`
|
||||
3. `bun run setup/ingest.ts` on a small fixture or targeted spec file if practical
|
||||
|
||||
### Behavior Cases To Verify
|
||||
|
||||
1. Returns one chunk when content is under `chunkSize`.
|
||||
2. Ignores HTML tag length when using cheerio text extraction.
|
||||
3. Does not split at separators unless size pressure requires it.
|
||||
4. Keeps protected nodes intact up to `maxChunkSize`.
|
||||
5. Recurses into protected nodes larger than `maxChunkSize`.
|
||||
6. Uses fallback splitting for oversized leaf text.
|
||||
7. Preserves metadata in `splitDocuments()`.
|
||||
8. Throws when `chunkOverlap !== 0`.
|
||||
9. Leaves hard-cap fallback disabled when `maxChunkSize` is omitted.
|
||||
|
||||
The implementation should also update `package.json` with a real `bun test` command so these tests can run in CI and locally.
|
||||
|
||||
### Example Important Cases
|
||||
|
||||
```typescript
|
||||
// separator is a soft hint, not an unconditional boundary
|
||||
const splitter = new HTMLTextSplitter({
|
||||
chunkSize: 100,
|
||||
maxChunkSize: 100,
|
||||
separators: ["h2"],
|
||||
});
|
||||
|
||||
await splitter.splitText("<div><h2>A</h2><p>short body</p></div>");
|
||||
// expected: one chunk because no size pressure exists
|
||||
```
|
||||
|
||||
```typescript
|
||||
// protected node stays atomic until maxChunkSize is exceeded
|
||||
const splitter = new HTMLTextSplitter({
|
||||
chunkSize: 20,
|
||||
maxChunkSize: 80,
|
||||
neverBreakWithin: ["pre"],
|
||||
});
|
||||
|
||||
await splitter.splitText("<pre>...60 chars...</pre>");
|
||||
// expected: one chunk
|
||||
```
|
||||
|
||||
## Repository Integration Notes
|
||||
|
||||
The eventual migration in `setup/ingest.ts` should account for these changes:
|
||||
|
||||
1. output will already be text chunks, so downstream `cheerio.load(chunk.pageContent).text()` cleanup may become redundant
|
||||
2. `keepSeparator` is removed from the new design
|
||||
3. `chunkOverlap` must remain `0`
|
||||
4. current warnings around chunk sizes should be rechecked after migration
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. No raw HTML offset slicing is used anywhere.
|
||||
2. The splitter traverses DOM structure recursively.
|
||||
3. Separators act as soft decomposition hints only.
|
||||
4. Protected nodes remain atomic unless `maxChunkSize` is exceeded.
|
||||
5. No output chunk exceeds `maxChunkSize` when a finite hard cap is configured.
|
||||
6. No overlap behavior is implemented or implied.
|
||||
7. `splitDocuments()` preserves metadata and returns chunked documents.
|
||||
8. The implementation can be verified in this repo without assuming a missing test framework.
|
||||
|
||||
## Open Questions
|
||||
|
||||
No remaining open questions in the current plan.
|
||||
Reference in New Issue
Block a user