ask262/.opencode/plans/1775291477517-lucky-garden.md

---
model: mimo-v2-pro
---

# Plan: HTML-to-Markdown Transformations for Ingestion

## Summary

Add configurable HTML-to-markdown transformations to `formatHTMLForIngestion.ts` to improve the quality of text stored in the vector index. Transformations convert ecmarkup/HTML elements (code blocks, cross-reference links, lists, inline code) to markdown syntax before the `HTMLTextSplitter` extracts plain text.

## Config Interface

```typescript
interface FormatConfig {
  codeBlocks: {
    block: string[];    // fenced code (pre>code, emu-eqn block)
    inline: string[];   // inline backticks (var, emu-val, emu-const, emu-eqn inline, code)
    grammar: string[];  // fenced bnf (emu-grammar)
  };
  links: string[];      // [text](#hash) (emu-xref a)
  lists: {
    ordered: string[];  // 1. 2. 3. (ol)
    unordered: string[];// - item (ul)
  };
  tables: string[];     // markdown tables (table, emu-table)
}
```

Default config:
```typescript
const DEFAULT_CONFIG: FormatConfig = {
  codeBlocks: {
    block: ["pre>code", "emu-eqn:not([class*='inline'])"],
    inline: ["var", "emu-val", "emu-const", "emu-eqn.inline"],
    // Note: standalone <code> (outside <pre>) is rare; handle with :not(pre > *) check in converter
    grammar: ["emu-grammar"],
  },
  links: ["emu-xref a"],
  lists: {
    ordered: ["ol"],
    unordered: ["ul"],
  },
  tables: ["table", "emu-table"],
};
```

## Transformation Order (leaf-to-parent)

1. **Inline leaves** (processed first, no children affected):
   - `emu-xref a[href]` → `<span class="link-markdown">[text](#hash)</span>`. Target the inner `<a>` tag (always has href, unlike `emu-xref` which may lack it).
     - Strip filename prefix from href (e.g., `abstract-operations.html#sec-tonumber` → `#sec-tonumber`)
     - Discard `e-user-code` class.
     - Replace `<a>` inside `emu-xref` (leaving wrapper `emu-xref` to be ignored by text splitter)
   - `var`, `emu-val`, `emu-const`, `code`, inline `emu-eqn` → `<span class="inline-code">`text`</span>`. Strip inner elements, plain text only.
2. **Block leaves**:
   - `pre>code` → `<pre class="code-markdown">```language\ntext\n```</pre>`. Language from `class` attribute. Strip hljs spans. Plain text.
   - block `emu-eqn` → `<pre class="code-markdown">```\ntext\n```</pre>`. No language tag. Strip inner elements.
3. **Structural parents** (children already resolved):
   - `emu-grammar` → `<pre class="code-markdown">```bnf\ntext\n```</pre>`. Extract text from grammar inner tags (emu-nt, emu-t, etc.).
   - `ol` → `<pre class="list-markdown">1. item\n   1. nested</pre>`. Nested via 2-space indentation.
   - `ul` → `<pre class="list-markdown">- item\n  - nested</pre>`. Nested via 2-space indentation.
   - `table`, `emu-table` → `<pre class="table-markdown">...existing markdown...</pre>`. Existing `convertTablesToMarkdown` works unchanged — `.text()` already extracts markdown text from pre-processed spans.
4. **`addNewlinesAfterBlocks($)`** (existing, unchanged — operates on remaining DOM elements)

## Implementation Details

### Approach: Cheerio only

All conversions use cheerio DOM manipulation (already in codebase). No external HTML-to-markdown library. Each conversion replaces DOM elements with text/markdown nodes. The existing `HTMLTextSplitter` processes the modified DOM unchanged.

### New functions in `setup/utils/formatHTMLForIngestion.ts`

Each function receives `cheerio.CheerioAPI` and modifies the DOM in-place.

1. **`convertLinksToMarkdown($, selector: string)`** — selector default: `emu-xref a[href]`
   - `$(selector).each(...)`: get href, strip filename, get text, replaceWith `<span>` containing `[text](#hash)`
   - Safety: skip if href starts with `http`

2. **`convertInlineCodeToMarkdown($, tags: string[])`**
   - `$(tag).each(...)`: get `.text()`, replaceWith `<span>` containing `` `text` ``
   - For `code` tags: skip if parent is `pre`

3. **`convertBlockCodeToMarkdown($, tags: string[])`**
   - For `pre>code`: get language from `class`, get `.text()` from code, replaceWith `<pre>` containing `` ```lang\ntext\n``` ``
   - For `emu-eqn` block: get `.text()`, replaceWith `<pre>` containing `` ```\ntext\n``` ``

4. **`convertGrammarToMarkdown($, selector: string)`**
   - Walk inner tags (emu-nt, emu-t, emu-rhs, emu-geq), build text representation
   - ReplaceWith `<pre>` containing `` ```bnf\ntext\n``` ``

5. **`convertListsToMarkdown($, ordered: string[], unordered: string[])`**
   - For each `ol`/`ul`: recursive function `listToMarkdown(elem, depth, isOrdered)`
   - Process `li` children, add `1. ` / `- ` prefix with indentation
   - ReplaceWith `<pre>` containing markdown text

6. **`formatForIngestion($, config?)`** — orchestrator, calls all in order

### Modify existing functions

- **`convertTablesToMarkdown($)`**: No changes needed. `.text()` already extracts markdown text from pre-processed `<span>` elements.

### Modify `setup/ingest.ts`

- Replace individual calls to `convertTablesToMarkdown($)` and `addNewlinesAfterBlocks($)` with single call to `formatForIngestion($)`.
- The child `emu-clause` replacement and deep nesting cleanup remain as-is (before `formatForIngestion`).

Updated call order in `buildSpecDocuments()`:
```typescript
// 1. Child clause replacement (existing)
$section.children("emu-clause").each(...);
$section.find("emu-clause").remove();

// 2. All formatting transformations (single call)
formatForIngestion($);
```

## Files to Modify

1. **`setup/utils/formatHTMLForIngestion.ts`** — add config interface, 5 new functions, orchestrating function
2. **`setup/ingest.ts`** — replace individual calls with `formatForIngestion($)` call
3. **New test file: `setup/utils/formatHTMLForIngestion.test.ts`** — unit tests for each transformation

No new dependencies. Cheerio only.

## Verification

1. **Unit tests**: Write tests for each transformation with sample HTML snippets from the spec:
   - `emu-xref a` → markdown link
   - `var`/`emu-val`/`emu-const` → inline code
   - `pre>code` → fenced code
   - `emu-grammar` → fenced bnf
   - `ol`/`ul` with nesting → markdown lists
   - `table` → markdown table (with links inside cells preserved)
   - Full pipeline via `formatForIngestion`
2. **Manual verification**: Run `bun run setup/ingest.ts` and inspect output chunks for correct markdown syntax
3. **Run lint/typecheck**: `bun run lint` and `bun run typecheck` (if available)