ask262/.opencode/plans/archive/1775113983739-sunny-knight.md

model

model
openai/gpt-5.4

HTMLTextSplitter Implementation Plan

Overview

Build a new HTMLTextSplitter that extends LangChain's TextSplitter and splits HTML by traversing the parsed DOM tree instead of slicing raw HTML strings.

This version is intentionally narrow:

• size metric uses cheerio text extraction only
• separators are CSS selectors
• neverBreakWithin is respected unless maxChunkSize forces a deeper split
• no chunkOverlap support
• output is text chunks, not reconstructed partial HTML

Confirmed Decisions

1. Use a traversal-based algorithm, not source offsets.
2. Only split when needed for size control.
3. Separator matches are soft structural hints, not unconditional split points.
4. If a separator subtree is too large, recurse into that subtree.
5. neverBreakWithin wins unless a node exceeds maxChunkSize; then recurse as a last resort.
6. Ignore separator matches inside protected subtrees during normal traversal.
7. V1 emits text chunks, not valid partial HTML.
8. Traverse both element nodes and text nodes.
9. Normalize output after boundary-aware joining so block-ish tags create readable spacing without adding synthetic spaces between inline tags.
10. When a direct child matches a separator during decomposition, it stays with the following content.
11. partIndex and totalParts are tracked per input document.
12. chunkOverlap is not supported.
13. Scope is the splitter itself, with ingest integration called out as a follow-up.

Goals

1. Extend TextSplitter from @langchain/textsplitters.
2. Accept HTML input and parse with cheerio.
3. Use CSS selectors for separators and neverBreakWithin.
4. Use chunkSize as a soft target and maxChunkSize as a hard limit.
5. Produce chunks whose content is text derived from the traversed HTML.
6. Recurse only when needed; avoid unnecessary fragmentation.

Non-Goals

1. Preserving exact HTML markup in chunk output.
2. Supporting chunkOverlap.
3. Building a full setup/ingest.ts migration in this plan.
4. Reproducing browser-accurate innerText behavior.

Public Interface

import { type Document } from "@langchain/core/documents";
import { TextSplitter, type TextSplitterParams } from "@langchain/textsplitters";

export interface HTMLTextSplitterParams extends TextSplitterParams {
  /**
   * CSS selectors that act as preferred structural split points.
   * They are only used when a subtree needs to be broken down.
   */
  separators?: string[];

  /**
   * CSS selectors for nodes that should stay atomic unless maxChunkSize forces recursion.
   */
  neverBreakWithin?: string[];

  /**
   * Absolute hard limit for chunk text length.
   * Defaults to Infinity if omitted.
   */
  maxChunkSize?: number;
}

export class HTMLTextSplitter extends TextSplitter {
  constructor(fields?: Partial<HTMLTextSplitterParams>);

  /**
   * Splits HTML input into normalized text chunks.
   *
   * Important: this splitter does not preserve HTML markup in output.
   * It uses parsed HTML structure for traversal and sizing, but emits text only.
   */
  splitText(text: string): Promise<string[]>;

  /**
   * Splits HTML documents into text-only child documents.
   *
   * Important: child document `pageContent` contains normalized text, not HTML.
   * Parent metadata is preserved and per-document part metadata is added.
   */
  splitDocuments(documents: Document[]): Promise<Document[]>;
}

Output Semantics

1. splitText() accepts HTML and returns string[] text chunks.
2. Each chunk is text content extracted from a DOM subtree or subtree fragment.
3. splitDocuments() preserves parent metadata and adds per-document partIndex and totalParts.
4. Final chunk text is normalized after DOM-aware joining; block-ish tag boundaries may add spacing, while inline tag boundaries do not add synthetic spaces.

Size Semantics

1. chunkSize is the target size.
2. maxChunkSize is the hard cap when provided.
3. The size function is based on the same normalized text content used for output segments.
4. Segment text is derived from cheerio text extraction, joined with DOM-aware boundaries, then whitespace-normalized before measurement.
5. No separate text-length implementation should be maintained outside that semantic.
6. If maxChunkSize is omitted, treat it as Infinity.

Core Algorithm

Mental Model

Treat each DOM node as a unit that can either:

• fit as-is
• stay atomic because it is protected and still within maxChunkSize
• be recursively decomposed into children because it is too large

Separators only matter when a node must be broken down. They influence recursion order; they do not force a split on their own.

Traversal Strategy

1. Parse the HTML with cheerio.
2. Select a traversal root:
   • use body children if a body exists
   • otherwise use root children
3. Walk top-level nodes in document order.
4. Convert each node into one or more text segments using recursive decomposition.
5. Merge resulting segments into final chunks while respecting chunkSize and maxChunkSize.

Recursive Decomposition Rules

For a node N:

1. Compute nodeText using the shared getNodeText() helper so both element nodes and text nodes follow the same trimming semantics.
2. If nodeText is empty, skip it.
3. If nodeText.length <= chunkSize, emit it as one segment.
4. If N matches neverBreakWithin and nodeText.length <= maxChunkSize, emit it as one segment.
5. Otherwise, recurse into children.
6. If N has no element/text children that can further subdivide the content, force-split the node text with RecursiveCharacterTextSplitter.

Separator-Aware Recursion

When recursing into a node with child elements:

1. Partition children into traversal groups.
2. Prefer group boundaries at direct child elements matching separators.
3. A separator child starts a new group and stays with the following sibling content rather than the preceding content.
4. If no separator child exists, recurse in plain DOM order.
5. Ignore separator matches inside any subtree currently treated as protected.

Internal Types

interface Segment {
  text: string;
  isProtected: boolean;
  sourceKind: "node" | "forced-split";
}

V1 should keep internal types local to the implementation file unless reuse appears during build.

Detailed Behavior

1. Protected Nodes

If a node matches neverBreakWithin, keep it atomic while its text length is <= maxChunkSize. Recurse into it only if it exceeds maxChunkSize.

If maxChunkSize is Infinity, protected nodes are never forced open by hard-cap rules.

2. Forced Splitting

When recursion reaches a node whose text still exceeds maxChunkSize, and there is no useful child structure left, use RecursiveCharacterTextSplitter on the extracted text.

Use it with:

• chunkSize: this.maxChunkSize
• chunkOverlap: 0

This is the final fallback that guarantees the hard limit.

If maxChunkSize is Infinity, this fallback is not used for hard-cap enforcement.

3. Merging

After recursion produces a flat list of text segments:

1. Start a new chunk.
2. Append the next segment if the combined length stays within chunkSize.
3. If appending would exceed chunkSize, flush the current chunk and start a new one.
4. If any individual segment is already larger than chunkSize but not larger than maxChunkSize, allow that segment to become its own chunk.
5. Join merged segment text using DOM-aware boundaries and whitespace normalization.
6. No overlap is applied.

When maxChunkSize is Infinity, a single segment may grow beyond chunkSize if it cannot be decomposed further without violating other rules.

Pseudocode

async splitText(html: string): Promise<string[]> {
  const $ = cheerio.load(html);
  const roots = $("body").length > 0 ? $("body").contents().toArray() : $.root().contents().toArray();

  const segments: Segment[] = [];
  for (const node of roots) {
    segments.push(...(await this.collectSegments($, node, false)));
  }

  return this.mergeSegments(segments);
}

private async collectSegments(
  $: cheerio.CheerioAPI,
  node: cheerio.AnyNode,
  inProtectedAncestor: boolean,
): Promise<Segment[]> {
  const text = this.getNodeText($, node);
  if (!text) return [];

  const isProtected = !inProtectedAncestor && this.matchesAny($, node, this.neverBreakWithin);

  if (text.length <= this.chunkSize) {
    return [{ text, isProtected, sourceKind: "node" }];
  }

  if (isProtected && text.length <= this.maxChunkSize) {
    return [{ text, isProtected, sourceKind: "node" }];
  }

  const childNodes = this.getChildNodesForRecursion(node);
  if (childNodes.length === 0) {
    return this.forceSplitText(text);
  }

  const childGroups = this.groupChildrenForDecomposition($, childNodes, isProtected || inProtectedAncestor);
  const segments: Segment[] = [];
  for (const group of childGroups) {
    for (const child of group) {
      segments.push(...(await this.collectSegments($, child, isProtected || inProtectedAncestor)));
    }
  }

  return segments.length > 0 ? segments : this.forceSplitText(text);
}

Utils Pseudocode

/**
 * Returns true when a tag node matches any configured CSS selector.
 *
 * Needed so separator and protected-node checks share the same selector logic
 * and non-tag nodes are ignored safely.
 */
private matchesAny(
  $: cheerio.CheerioAPI,
  node: cheerio.AnyNode,
  selectors: string[],
): boolean {
  if (node.type !== "tag") {
    return false;
  }

  for (const selector of selectors) {
    if ($(node).is(selector)) {
      return true;
    }
  }

  return false;
}

/**
 * Extracts the canonical text value for a node.
 *
 * Needed so sizing and emitted output use the same trimmed text semantics for
 * both element nodes and raw text nodes.
 */
private getNodeText($: cheerio.CheerioAPI, node: cheerio.AnyNode): string {
  if (node.type === "text") {
    return (node.data ?? "").trim();
  }

  return $(node).text().trim();
}

/**
 * Returns the child nodes that are meaningful recursion units.
 *
 * Needed to recurse through DOM structure without carrying empty whitespace-only
 * text nodes or unsupported node types through the algorithm.
 */
private getChildNodesForRecursion(node: cheerio.AnyNode): cheerio.AnyNode[] {
  if (!("children" in node) || !Array.isArray(node.children)) {
    return [];
  }

  return node.children.filter((child) => {
    if (child.type === "text") {
      return (child.data ?? "").trim().length > 0;
    }

    return child.type === "tag";
  });
}

/**
 * Groups child nodes so separator nodes start a new group.
 *
 * Needed because separators represent preferred section boundaries. Grouping
 * makes "separator plus following content" explicit during recursion.
 */
private groupChildrenForDecomposition(
  $: cheerio.CheerioAPI,
  childNodes: cheerio.AnyNode[],
  inProtectedTree: boolean,
): cheerio.AnyNode[][] {
  if (inProtectedTree || this.separators.length === 0) {
    return [childNodes];
  }

  const groups: cheerio.AnyNode[][] = [];
  let currentGroup: cheerio.AnyNode[] = [];

  const flush = () => {
    if (currentGroup.length > 0) {
      groups.push(currentGroup);
      currentGroup = [];
    }
  };

  for (const child of childNodes) {
    if (this.matchesAny($, child, this.separators)) {
      flush();
    }

    currentGroup.push(child);
  }

  flush();
  return groups;
}

/**
 * Applies the final plain-text fallback when a node cannot be decomposed further.
 *
 * Needed to enforce a finite hard cap with `RecursiveCharacterTextSplitter`
 * once DOM structure is exhausted.
 */
private async forceSplitText(text: string): Promise<Segment[]> {
  if (!Number.isFinite(this.maxChunkSize)) {
    return [{ text: text.trim(), isProtected: false, sourceKind: "forced-split" }];
  }

  const fallback = new RecursiveCharacterTextSplitter({
    chunkSize: this.maxChunkSize,
    chunkOverlap: 0,
  });

  const chunks = await fallback.splitText(text);
  return chunks
    .map((chunk) => chunk.trim())
    .filter(Boolean)
    .map((chunk) => ({
      text: chunk,
      isProtected: false,
      sourceKind: "forced-split" as const,
    }));
}

/**
 * Merges flat segments into final chunks using `chunkSize` as a soft target.
 *
 * Needed to keep chunk assembly separate from DOM traversal and to normalize the
 * final text output by joining segment text with single spaces.
 */
private mergeSegments(segments: Segment[]): string[] {
  const chunks: string[] = [];
  let currentParts: string[] = [];
  let currentLength = 0;

  const flush = () => {
    if (currentParts.length === 0) {
      return;
    }

    chunks.push(currentParts.join(" "));
    currentParts = [];
    currentLength = 0;
  };

  for (const segment of segments) {
    const nextLength = currentLength === 0
      ? segment.text.length
      : currentLength + 1 + segment.text.length;

    if (currentParts.length > 0 && nextLength > this.chunkSize) {
      flush();
    }

    currentParts.push(segment.text);
    currentLength = currentLength === 0
      ? segment.text.length
      : currentLength + 1 + segment.text.length;
  }

  flush();
  return chunks;
}

Implementation Plan

Phase 1: Class Skeleton

1. Create textsplitters/html_text_splitter.ts.
2. Define HTMLTextSplitterParams.
3. Extend TextSplitter.
4. Reject unsupported overlap in the constructor:
   • if chunkOverlap !== 0, throw a clear error
5. Default maxChunkSize to Infinity when omitted.

Phase 2: Tree Traversal Helpers

1. matchesAny() for selector matching.
2. getNodeText() using cheerio text extraction and trimming semantics shared with output.
3. getChildNodesForRecursion() to expose both text and element children in DOM order.
4. groupChildrenForDecomposition() to build explicit child groups, with separator nodes grouped with following content.

Phase 3: Recursive Collection

1. Implement collectSegments().
2. Preserve document order.
3. Skip empty/whitespace-only results.

Phase 4: Final Fallback

1. Implement forceSplitText() with RecursiveCharacterTextSplitter.
2. Use maxChunkSize as the chunk size when it is finite.
3. Use chunkOverlap: 0.
4. Trim and drop empty results.

Phase 5: Merge And Document Support

1. Merge segments into final chunks using chunkSize as the soft target.
2. Implement splitDocuments() by mapping each input document through splitText().
3. Preserve original metadata and add per-document partIndex / totalParts.

Edge Cases

Empty HTML

Return [].

Plain Text Input

Cheerio will still parse it; return one or more text chunks based on size.

Deeply Nested Protected Content

Keep it atomic unless it exceeds maxChunkSize; then recurse into its children and only force-split as a last resort.

If maxChunkSize is Infinity, it remains atomic unless some ancestor decomposition requires visiting its children.

Separator Inside Protected Content

Ignore the separator during normal traversal.

Large Text Node With No Useful Child Structure

Use text fallback splitting.

Consecutive Small Nodes

Merge them up to chunkSize.

A Single Segment Between chunkSize And maxChunkSize

Allow it as a standalone chunk.

Testing Strategy

Use Bun's built-in test runner and add automated tests under test/textsplitters/.

Minimum Verification

1. bunx tsc --noEmit
2. automated splitter tests under test/textsplitters/
3. bun run setup/ingest.ts on a small fixture or targeted spec file if practical

Behavior Cases To Verify

1. Returns one chunk when content is under chunkSize.
2. Ignores HTML tag length when using cheerio text extraction.
3. Does not split at separators unless size pressure requires it.
4. Keeps protected nodes intact up to maxChunkSize.
5. Recurses into protected nodes larger than maxChunkSize.
6. Uses fallback splitting for oversized leaf text.
7. Preserves metadata in splitDocuments().
8. Throws when chunkOverlap !== 0.
9. Leaves hard-cap fallback disabled when maxChunkSize is omitted.

The implementation should also update package.json with a real bun test command so these tests can run in CI and locally.

Example Important Cases

// separator is a soft hint, not an unconditional boundary
const splitter = new HTMLTextSplitter({
  chunkSize: 100,
  maxChunkSize: 100,
  separators: ["h2"],
});

await splitter.splitText("<div><h2>A</h2><p>short body</p></div>");
// expected: one chunk because no size pressure exists

// protected node stays atomic until maxChunkSize is exceeded
const splitter = new HTMLTextSplitter({
  chunkSize: 20,
  maxChunkSize: 80,
  neverBreakWithin: ["pre"],
});

await splitter.splitText("<pre>...60 chars...</pre>");
// expected: one chunk

Repository Integration Notes

The eventual migration in setup/ingest.ts should account for these changes:

1. output will already be text chunks, so downstream cheerio.load(chunk.pageContent).text() cleanup may become redundant
2. keepSeparator is removed from the new design
3. chunkOverlap must remain 0
4. current warnings around chunk sizes should be rechecked after migration

Success Criteria

1. No raw HTML offset slicing is used anywhere.
2. The splitter traverses DOM structure recursively.
3. Separators act as soft decomposition hints only.
4. Protected nodes remain atomic unless maxChunkSize is exceeded.
5. No output chunk exceeds maxChunkSize when a finite hard cap is configured.
6. No overlap behavior is implemented or implied.
7. splitDocuments() preserves metadata and returns chunked documents.
8. The implementation can be verified in this repo without assuming a missing test framework.

Open Questions

No remaining open questions in the current plan.