ask262/.opencode/plans/archive/1775113983739-sunny-knight.md

---
model: openai/gpt-5.4
---

# HTMLTextSplitter Implementation Plan

## Overview

Build a new `HTMLTextSplitter` that extends LangChain's `TextSplitter` and splits HTML by traversing the parsed DOM tree instead of slicing raw HTML strings.

This version is intentionally narrow:
- size metric uses cheerio text extraction only
- separators are CSS selectors
- `neverBreakWithin` is respected unless `maxChunkSize` forces a deeper split
- no `chunkOverlap` support
- output is text chunks, not reconstructed partial HTML

## Confirmed Decisions

1. Use a traversal-based algorithm, not source offsets.
2. Only split when needed for size control.
3. Separator matches are soft structural hints, not unconditional split points.
4. If a separator subtree is too large, recurse into that subtree.
5. `neverBreakWithin` wins unless a node exceeds `maxChunkSize`; then recurse as a last resort.
6. Ignore separator matches inside protected subtrees during normal traversal.
7. V1 emits text chunks, not valid partial HTML.
8. Traverse both element nodes and text nodes.
9. Normalize output after boundary-aware joining so block-ish tags create readable spacing without adding synthetic spaces between inline tags.
10. When a direct child matches a separator during decomposition, it stays with the following content.
11. `partIndex` and `totalParts` are tracked per input document.
12. `chunkOverlap` is not supported.
13. Scope is the splitter itself, with ingest integration called out as a follow-up.

## Goals

1. Extend `TextSplitter` from `@langchain/textsplitters`.
2. Accept HTML input and parse with `cheerio`.
3. Use CSS selectors for `separators` and `neverBreakWithin`.
4. Use `chunkSize` as a soft target and `maxChunkSize` as a hard limit.
5. Produce chunks whose content is text derived from the traversed HTML.
6. Recurse only when needed; avoid unnecessary fragmentation.

## Non-Goals

1. Preserving exact HTML markup in chunk output.
2. Supporting `chunkOverlap`.
3. Building a full `setup/ingest.ts` migration in this plan.
4. Reproducing browser-accurate `innerText` behavior.

## Public Interface

```typescript
import { type Document } from "@langchain/core/documents";
import { TextSplitter, type TextSplitterParams } from "@langchain/textsplitters";

export interface HTMLTextSplitterParams extends TextSplitterParams {
  /**
   * CSS selectors that act as preferred structural split points.
   * They are only used when a subtree needs to be broken down.
   */
  separators?: string[];

  /**
   * CSS selectors for nodes that should stay atomic unless maxChunkSize forces recursion.
   */
  neverBreakWithin?: string[];

  /**
   * Absolute hard limit for chunk text length.
   * Defaults to Infinity if omitted.
   */
  maxChunkSize?: number;
}

export class HTMLTextSplitter extends TextSplitter {
  constructor(fields?: Partial<HTMLTextSplitterParams>);

  /**
   * Splits HTML input into normalized text chunks.
   *
   * Important: this splitter does not preserve HTML markup in output.
   * It uses parsed HTML structure for traversal and sizing, but emits text only.
   */
  splitText(text: string): Promise<string[]>;

  /**
   * Splits HTML documents into text-only child documents.
   *
   * Important: child document `pageContent` contains normalized text, not HTML.
   * Parent metadata is preserved and per-document part metadata is added.
   */
  splitDocuments(documents: Document[]): Promise<Document[]>;
}
```

## Output Semantics

1. `splitText()` accepts HTML and returns `string[]` text chunks.
2. Each chunk is text content extracted from a DOM subtree or subtree fragment.
3. `splitDocuments()` preserves parent metadata and adds per-document `partIndex` and `totalParts`.
4. Final chunk text is normalized after DOM-aware joining; block-ish tag boundaries may add spacing, while inline tag boundaries do not add synthetic spaces.

## Size Semantics

1. `chunkSize` is the target size.
2. `maxChunkSize` is the hard cap when provided.
3. The size function is based on the same normalized text content used for output segments.
4. Segment text is derived from cheerio text extraction, joined with DOM-aware boundaries, then whitespace-normalized before measurement.
5. No separate text-length implementation should be maintained outside that semantic.
6. If `maxChunkSize` is omitted, treat it as `Infinity`.

## Core Algorithm

### Mental Model

Treat each DOM node as a unit that can either:
- fit as-is
- stay atomic because it is protected and still within `maxChunkSize`
- be recursively decomposed into children because it is too large

Separators only matter when a node must be broken down. They influence recursion order; they do not force a split on their own.

### Traversal Strategy

1. Parse the HTML with cheerio.
2. Select a traversal root:
   - use `body` children if a body exists
   - otherwise use root children
3. Walk top-level nodes in document order.
4. Convert each node into one or more text segments using recursive decomposition.
5. Merge resulting segments into final chunks while respecting `chunkSize` and `maxChunkSize`.

### Recursive Decomposition Rules

For a node `N`:

1. Compute `nodeText` using the shared `getNodeText()` helper so both element nodes and text nodes follow the same trimming semantics.
2. If `nodeText` is empty, skip it.
3. If `nodeText.length <= chunkSize`, emit it as one segment.
4. If `N` matches `neverBreakWithin` and `nodeText.length <= maxChunkSize`, emit it as one segment.
5. Otherwise, recurse into children.
6. If `N` has no element/text children that can further subdivide the content, force-split the node text with `RecursiveCharacterTextSplitter`.

### Separator-Aware Recursion

When recursing into a node with child elements:

1. Partition children into traversal groups.
2. Prefer group boundaries at direct child elements matching `separators`.
3. A separator child starts a new group and stays with the following sibling content rather than the preceding content.
4. If no separator child exists, recurse in plain DOM order.
5. Ignore separator matches inside any subtree currently treated as protected.

## Internal Types

```typescript
interface Segment {
  text: string;
  isProtected: boolean;
  sourceKind: "node" | "forced-split";
}
```

V1 should keep internal types local to the implementation file unless reuse appears during build.

## Detailed Behavior

### 1. Protected Nodes

If a node matches `neverBreakWithin`, keep it atomic while its text length is `<= maxChunkSize`. Recurse into it only if it exceeds `maxChunkSize`.

If `maxChunkSize` is `Infinity`, protected nodes are never forced open by hard-cap rules.

### 2. Forced Splitting

When recursion reaches a node whose text still exceeds `maxChunkSize`, and there is no useful child structure left, use `RecursiveCharacterTextSplitter` on the extracted text.

Use it with:
- `chunkSize: this.maxChunkSize`
- `chunkOverlap: 0`

This is the final fallback that guarantees the hard limit.

If `maxChunkSize` is `Infinity`, this fallback is not used for hard-cap enforcement.

### 3. Merging

After recursion produces a flat list of text segments:

1. Start a new chunk.
2. Append the next segment if the combined length stays within `chunkSize`.
3. If appending would exceed `chunkSize`, flush the current chunk and start a new one.
4. If any individual segment is already larger than `chunkSize` but not larger than `maxChunkSize`, allow that segment to become its own chunk.
5. Join merged segment text using DOM-aware boundaries and whitespace normalization.
6. No overlap is applied.

When `maxChunkSize` is `Infinity`, a single segment may grow beyond `chunkSize` if it cannot be decomposed further without violating other rules.

## Pseudocode

```typescript
async splitText(html: string): Promise<string[]> {
  const $ = cheerio.load(html);
  const roots = $("body").length > 0 ? $("body").contents().toArray() : $.root().contents().toArray();

  const segments: Segment[] = [];
  for (const node of roots) {
    segments.push(...(await this.collectSegments($, node, false)));
  }

  return this.mergeSegments(segments);
}

private async collectSegments(
  $: cheerio.CheerioAPI,
  node: cheerio.AnyNode,
  inProtectedAncestor: boolean,
): Promise<Segment[]> {
  const text = this.getNodeText($, node);
  if (!text) return [];

  const isProtected = !inProtectedAncestor && this.matchesAny($, node, this.neverBreakWithin);

  if (text.length <= this.chunkSize) {
    return [{ text, isProtected, sourceKind: "node" }];
  }

  if (isProtected && text.length <= this.maxChunkSize) {
    return [{ text, isProtected, sourceKind: "node" }];
  }

  const childNodes = this.getChildNodesForRecursion(node);
  if (childNodes.length === 0) {
    return this.forceSplitText(text);
  }

  const childGroups = this.groupChildrenForDecomposition($, childNodes, isProtected || inProtectedAncestor);
  const segments: Segment[] = [];
  for (const group of childGroups) {
    for (const child of group) {
      segments.push(...(await this.collectSegments($, child, isProtected || inProtectedAncestor)));
    }
  }

  return segments.length > 0 ? segments : this.forceSplitText(text);
}
```

## Utils Pseudocode

```typescript
/**
 * Returns true when a tag node matches any configured CSS selector.
 *
 * Needed so separator and protected-node checks share the same selector logic
 * and non-tag nodes are ignored safely.
 */
private matchesAny(
  $: cheerio.CheerioAPI,
  node: cheerio.AnyNode,
  selectors: string[],
): boolean {
  if (node.type !== "tag") {
    return false;
  }

  for (const selector of selectors) {
    if ($(node).is(selector)) {
      return true;
    }
  }

  return false;
}

/**
 * Extracts the canonical text value for a node.
 *
 * Needed so sizing and emitted output use the same trimmed text semantics for
 * both element nodes and raw text nodes.
 */
private getNodeText($: cheerio.CheerioAPI, node: cheerio.AnyNode): string {
  if (node.type === "text") {
    return (node.data ?? "").trim();
  }

  return $(node).text().trim();
}

/**
 * Returns the child nodes that are meaningful recursion units.
 *
 * Needed to recurse through DOM structure without carrying empty whitespace-only
 * text nodes or unsupported node types through the algorithm.
 */
private getChildNodesForRecursion(node: cheerio.AnyNode): cheerio.AnyNode[] {
  if (!("children" in node) || !Array.isArray(node.children)) {
    return [];
  }

  return node.children.filter((child) => {
    if (child.type === "text") {
      return (child.data ?? "").trim().length > 0;
    }

    return child.type === "tag";
  });
}

/**
 * Groups child nodes so separator nodes start a new group.
 *
 * Needed because separators represent preferred section boundaries. Grouping
 * makes "separator plus following content" explicit during recursion.
 */
private groupChildrenForDecomposition(
  $: cheerio.CheerioAPI,
  childNodes: cheerio.AnyNode[],
  inProtectedTree: boolean,
): cheerio.AnyNode[][] {
  if (inProtectedTree || this.separators.length === 0) {
    return [childNodes];
  }

  const groups: cheerio.AnyNode[][] = [];
  let currentGroup: cheerio.AnyNode[] = [];

  const flush = () => {
    if (currentGroup.length > 0) {
      groups.push(currentGroup);
      currentGroup = [];
    }
  };

  for (const child of childNodes) {
    if (this.matchesAny($, child, this.separators)) {
      flush();
    }

    currentGroup.push(child);
  }

  flush();
  return groups;
}

/**
 * Applies the final plain-text fallback when a node cannot be decomposed further.
 *
 * Needed to enforce a finite hard cap with `RecursiveCharacterTextSplitter`
 * once DOM structure is exhausted.
 */
private async forceSplitText(text: string): Promise<Segment[]> {
  if (!Number.isFinite(this.maxChunkSize)) {
    return [{ text: text.trim(), isProtected: false, sourceKind: "forced-split" }];
  }

  const fallback = new RecursiveCharacterTextSplitter({
    chunkSize: this.maxChunkSize,
    chunkOverlap: 0,
  });

  const chunks = await fallback.splitText(text);
  return chunks
    .map((chunk) => chunk.trim())
    .filter(Boolean)
    .map((chunk) => ({
      text: chunk,
      isProtected: false,
      sourceKind: "forced-split" as const,
    }));
}

/**
 * Merges flat segments into final chunks using `chunkSize` as a soft target.
 *
 * Needed to keep chunk assembly separate from DOM traversal and to normalize the
 * final text output by joining segment text with single spaces.
 */
private mergeSegments(segments: Segment[]): string[] {
  const chunks: string[] = [];
  let currentParts: string[] = [];
  let currentLength = 0;

  const flush = () => {
    if (currentParts.length === 0) {
      return;
    }

    chunks.push(currentParts.join(" "));
    currentParts = [];
    currentLength = 0;
  };

  for (const segment of segments) {
    const nextLength = currentLength === 0
      ? segment.text.length
      : currentLength + 1 + segment.text.length;

    if (currentParts.length > 0 && nextLength > this.chunkSize) {
      flush();
    }

    currentParts.push(segment.text);
    currentLength = currentLength === 0
      ? segment.text.length
      : currentLength + 1 + segment.text.length;
  }

  flush();
  return chunks;
}
```

## Implementation Plan

### Phase 1: Class Skeleton

1. Create `textsplitters/html_text_splitter.ts`.
2. Define `HTMLTextSplitterParams`.
3. Extend `TextSplitter`.
4. Reject unsupported overlap in the constructor:
   - if `chunkOverlap !== 0`, throw a clear error
5. Default `maxChunkSize` to `Infinity` when omitted.

### Phase 2: Tree Traversal Helpers

1. `matchesAny()` for selector matching.
2. `getNodeText()` using cheerio text extraction and trimming semantics shared with output.
3. `getChildNodesForRecursion()` to expose both text and element children in DOM order.
4. `groupChildrenForDecomposition()` to build explicit child groups, with separator nodes grouped with following content.

### Phase 3: Recursive Collection

1. Implement `collectSegments()`.
2. Preserve document order.
3. Skip empty/whitespace-only results.

### Phase 4: Final Fallback

1. Implement `forceSplitText()` with `RecursiveCharacterTextSplitter`.
2. Use `maxChunkSize` as the chunk size when it is finite.
3. Use `chunkOverlap: 0`.
4. Trim and drop empty results.

### Phase 5: Merge And Document Support

1. Merge segments into final chunks using `chunkSize` as the soft target.
2. Implement `splitDocuments()` by mapping each input document through `splitText()`.
3. Preserve original metadata and add per-document `partIndex` / `totalParts`.

## Edge Cases

### Empty HTML

Return `[]`.

### Plain Text Input

Cheerio will still parse it; return one or more text chunks based on size.

### Deeply Nested Protected Content

Keep it atomic unless it exceeds `maxChunkSize`; then recurse into its children and only force-split as a last resort.

If `maxChunkSize` is `Infinity`, it remains atomic unless some ancestor decomposition requires visiting its children.

### Separator Inside Protected Content

Ignore the separator during normal traversal.

### Large Text Node With No Useful Child Structure

Use text fallback splitting.

### Consecutive Small Nodes

Merge them up to `chunkSize`.

### A Single Segment Between chunkSize And maxChunkSize

Allow it as a standalone chunk.

## Testing Strategy

Use Bun's built-in test runner and add automated tests under `test/textsplitters/`.

### Minimum Verification

1. `bunx tsc --noEmit`
2. automated splitter tests under `test/textsplitters/`
3. `bun run setup/ingest.ts` on a small fixture or targeted spec file if practical

### Behavior Cases To Verify

1. Returns one chunk when content is under `chunkSize`.
2. Ignores HTML tag length when using cheerio text extraction.
3. Does not split at separators unless size pressure requires it.
4. Keeps protected nodes intact up to `maxChunkSize`.
5. Recurses into protected nodes larger than `maxChunkSize`.
6. Uses fallback splitting for oversized leaf text.
7. Preserves metadata in `splitDocuments()`.
8. Throws when `chunkOverlap !== 0`.
9. Leaves hard-cap fallback disabled when `maxChunkSize` is omitted.

The implementation should also update `package.json` with a real `bun test` command so these tests can run in CI and locally.

### Example Important Cases

```typescript
// separator is a soft hint, not an unconditional boundary
const splitter = new HTMLTextSplitter({
  chunkSize: 100,
  maxChunkSize: 100,
  separators: ["h2"],
});

await splitter.splitText("<div><h2>A</h2><p>short body</p></div>");
// expected: one chunk because no size pressure exists
```

```typescript
// protected node stays atomic until maxChunkSize is exceeded
const splitter = new HTMLTextSplitter({
  chunkSize: 20,
  maxChunkSize: 80,
  neverBreakWithin: ["pre"],
});

await splitter.splitText("<pre>...60 chars...</pre>");
// expected: one chunk
```

## Repository Integration Notes

The eventual migration in `setup/ingest.ts` should account for these changes:

1. output will already be text chunks, so downstream `cheerio.load(chunk.pageContent).text()` cleanup may become redundant
2. `keepSeparator` is removed from the new design
3. `chunkOverlap` must remain `0`
4. current warnings around chunk sizes should be rechecked after migration

## Success Criteria

1. No raw HTML offset slicing is used anywhere.
2. The splitter traverses DOM structure recursively.
3. Separators act as soft decomposition hints only.
4. Protected nodes remain atomic unless `maxChunkSize` is exceeded.
5. No output chunk exceeds `maxChunkSize` when a finite hard cap is configured.
6. No overlap behavior is implemented or implied.
7. `splitDocuments()` preserves metadata and returns chunked documents.
8. The implementation can be verified in this repo without assuming a missing test framework.

## Open Questions

No remaining open questions in the current plan.