ask262/.opencode/plans/1775291477517-lucky-garden.md

model

model
mimo-v2-pro

Plan: HTML-to-Markdown Transformations for Ingestion

Summary

Add configurable HTML-to-markdown transformations to formatHTMLForIngestion.ts to improve the quality of text stored in the vector index. Transformations convert ecmarkup/HTML elements (code blocks, cross-reference links, lists, inline code) to markdown syntax before the HTMLTextSplitter extracts plain text.

Config Interface

interface FormatConfig {
  codeBlocks: {
    block: string[];    // fenced code (pre>code, emu-eqn block)
    inline: string[];   // inline backticks (var, emu-val, emu-const, emu-eqn inline, code)
    grammar: string[];  // fenced bnf (emu-grammar)
  };
  links: string[];      // [text](#hash) (emu-xref a)
  lists: {
    ordered: string[];  // 1. 2. 3. (ol)
    unordered: string[];// - item (ul)
  };
  tables: string[];     // markdown tables (table, emu-table)
}

Default config:

const DEFAULT_CONFIG: FormatConfig = {
  codeBlocks: {
    block: ["pre>code", "emu-eqn:not([class*='inline'])"],
    inline: ["var", "emu-val", "emu-const", "emu-eqn.inline"],
    // Note: standalone <code> (outside <pre>) is rare; handle with :not(pre > *) check in converter
    grammar: ["emu-grammar"],
  },
  links: ["emu-xref a"],
  lists: {
    ordered: ["ol"],
    unordered: ["ul"],
  },
  tables: ["table", "emu-table"],
};

Transformation Order (leaf-to-parent)

1. Inline leaves (processed first, no children affected):
   • emu-xref a[href] → <span class="link-markdown">[text](#hash)</span>. Target the inner <a> tag (always has href, unlike emu-xref which may lack it).
     • Strip filename prefix from href (e.g., abstract-operations.html#sec-tonumber → #sec-tonumber)
     • Discard e-user-code class.
     • Replace <a> inside emu-xref (leaving wrapper emu-xref to be ignored by text splitter)
   • var, emu-val, emu-const, code, inline emu-eqn → <span class="inline-code">text</span>. Strip inner elements, plain text only.
2. Block leaves:
   • pre>code → <pre class="code-markdown">```language\ntext\n```</pre>. Language from class attribute. Strip hljs spans. Plain text.
   • block emu-eqn → <pre class="code-markdown">```\ntext\n```</pre>. No language tag. Strip inner elements.
3. Structural parents (children already resolved):
   • emu-grammar → <pre class="code-markdown">```bnf\ntext\n```</pre>. Extract text from grammar inner tags (emu-nt, emu-t, etc.).
   • ol → <pre class="list-markdown">1. item\n 1. nested</pre>. Nested via 2-space indentation.
   • ul → <pre class="list-markdown">- item\n - nested</pre>. Nested via 2-space indentation.
   • table, emu-table → <pre class="table-markdown">...existing markdown...</pre>. Existing convertTablesToMarkdown works unchanged — .text() already extracts markdown text from pre-processed spans.
4. addNewlinesAfterBlocks($) (existing, unchanged — operates on remaining DOM elements)

Implementation Details

Approach: Cheerio only

All conversions use cheerio DOM manipulation (already in codebase). No external HTML-to-markdown library. Each conversion replaces DOM elements with text/markdown nodes. The existing HTMLTextSplitter processes the modified DOM unchanged.

New functions in setup/utils/formatHTMLForIngestion.ts

Each function receives cheerio.CheerioAPI and modifies the DOM in-place.

1. convertLinksToMarkdown($, selector: string) — selector default: emu-xref a[href]

   • $(selector).each(...): get href, strip filename, get text, replaceWith <span> containing [text](#hash)
   • Safety: skip if href starts with http

2. convertInlineCodeToMarkdown($, tags: string[])

   • $(tag).each(...): get .text(), replaceWith <span> containing `text`
   • For code tags: skip if parent is pre

3. convertBlockCodeToMarkdown($, tags: string[])

   • For pre>code: get language from class, get .text() from code, replaceWith <pre> containing ```lang\ntext\n```
   • For emu-eqn block: get .text(), replaceWith <pre> containing ```\ntext\n```

4. convertGrammarToMarkdown($, selector: string)

   • Walk inner tags (emu-nt, emu-t, emu-rhs, emu-geq), build text representation
   • ReplaceWith <pre> containing ```bnf\ntext\n```

5. convertListsToMarkdown($, ordered: string[], unordered: string[])

   • For each ol/ul: recursive function listToMarkdown(elem, depth, isOrdered)
   • Process li children, add 1. / - prefix with indentation
   • ReplaceWith <pre> containing markdown text

6. formatForIngestion($, config?) — orchestrator, calls all in order

Modify existing functions

• convertTablesToMarkdown($): No changes needed. .text() already extracts markdown text from pre-processed <span> elements.

Modify setup/ingest.ts

• Replace individual calls to convertTablesToMarkdown($) and addNewlinesAfterBlocks($) with single call to formatForIngestion($).
• The child emu-clause replacement and deep nesting cleanup remain as-is (before formatForIngestion).

Updated call order in buildSpecDocuments():

// 1. Child clause replacement (existing)
$section.children("emu-clause").each(...);
$section.find("emu-clause").remove();

// 2. All formatting transformations (single call)
formatForIngestion($);

Files to Modify

1. setup/utils/formatHTMLForIngestion.ts — add config interface, 5 new functions, orchestrating function
2. setup/ingest.ts — replace individual calls with formatForIngestion($) call
3. New test file: setup/utils/formatHTMLForIngestion.test.ts — unit tests for each transformation

No new dependencies. Cheerio only.

Verification

1. Unit tests: Write tests for each transformation with sample HTML snippets from the spec:
   • emu-xref a → markdown link
   • var/emu-val/emu-const → inline code
   • pre>code → fenced code
   • emu-grammar → fenced bnf
   • ol/ul with nesting → markdown lists
   • table → markdown table (with links inside cells preserved)
   • Full pipeline via formatForIngestion
2. Manual verification: Run bun run setup/ingest.ts and inspect output chunks for correct markdown syntax
3. Run lint/typecheck: bun run lint and bun run typecheck (if available)