` containing `` `text` ``
- For `code` tags: skip if parent is `pre`
3. **`convertBlockCodeToMarkdown($, tags: string[])`**
- For `pre>code`: get language from `class`, get `.text()` from code, replaceWith `` containing `` ```lang\ntext\n``` ``
- For `emu-eqn` block: get `.text()`, replaceWith `` containing `` ```\ntext\n``` ``
4. **`convertGrammarToMarkdown($, selector: string)`**
- Walk inner tags (emu-nt, emu-t, emu-rhs, emu-geq), build text representation
- ReplaceWith `` containing `` ```bnf\ntext\n``` ``
5. **`convertListsToMarkdown($, ordered: string[], unordered: string[])`**
- For each `ol`/`ul`: recursive function `listToMarkdown(elem, depth, isOrdered)`
- Process `li` children, add `1. ` / `- ` prefix with indentation
- ReplaceWith `` containing markdown text
6. **`formatForIngestion($, config?)`** — orchestrator, calls all in order
## Critical Implementation Code
### Config interface
```typescript
export interface FormatConfig {
codeBlocks: {
block: string[];
inline: string[];
grammar: string[];
};
links: string[];
lists: {
ordered: string[];
unordered: string[];
};
tables: string[];
}
export const DEFAULT_CONFIG: FormatConfig = {
codeBlocks: {
block: ["pre>code", "emu-eqn:not([class*='inline'])"],
inline: ["var", "emu-val", "emu-const", "emu-eqn.inline"],
grammar: ["emu-grammar"],
},
links: ["emu-xref a"],
lists: {
ordered: ["ol"],
unordered: ["ul"],
},
tables: ["table", "emu-table"],
};
```
### `convertLinksToMarkdown` — filename stripping
```typescript
export function convertLinksToMarkdown(
$: cheerio.CheerioAPI,
selector: string = "emu-xref a[href]",
): void {
$(selector).each((_, elem) => {
const $a = $(elem);
const href = $a.attr("href") ?? "";
// Skip external links
if (href.startsWith("http")) return;
// Strip filename prefix: "abstract-operations.html#sec-tonumber" → "#sec-tonumber"
const hash = href.includes("#") ? `#${href.split("#")[1]}` : href;
const text = $a.text().trim();
if (!text || !hash) return;
$a.replaceWith(`[${text}](${hash})`);
});
}
```
### `convertInlineCodeToMarkdown` — backtick wrapping
```typescript
export function convertInlineCodeToMarkdown(
$: cheerio.CheerioAPI,
tags: string[],
): void {
for (const tag of tags) {
$(tag).each((_, elem) => {
// Skip inside (handled by block converter)
if (elem.name === "code" && $(elem).parent("pre").length > 0) return;
const text = $(elem).text().trim();
if (!text) return;
$(elem).replaceWith(`\`${text}\``);
});
}
}
```
### `convertBlockCodeToMarkdown` — fenced code blocks
```typescript
function extractLanguage(codeClass: string | undefined): string {
// "javascript hljs" → "javascript", "python hljs" → "python"
if (!codeClass) return "";
const match = codeClass.match(/^(\w+)/);
return match?.[1] ?? "";
}
export function convertBlockCodeToMarkdown(
$: cheerio.CheerioAPI,
tags: string[],
): void {
// Handle pre>code (JS code examples)
$("pre > code").each((_, code) => {
const lang = extractLanguage($(code).attr("class"));
const text = $(code).text().trim();
const pre = $(code).parent("pre");
pre.replaceWith(
$(`\`\`\`${lang}\n${text}\n\`\`\``),
);
});
// Handle standalone emu-eqn (block-level equations)
$("emu-eqn:not([class*='inline'])").each((_, eqn) => {
const text = $(eqn).text().trim();
$(eqn).replaceWith(
$(`\`\`\`\n${text}\n\`\`\``),
);
});
}
```
### `convertListsToMarkdown` — recursive list processing
```typescript
function listToMarkdown(
$: cheerio.CheerioAPI,
elem: cheerio.Element,
depth: number = 0,
): string {
const $elem = $(elem);
const isOrdered = elem.name === "ol";
const indent = " ".repeat(depth);
const lines: string[] = [];
$elem.children("li").each((i, li) => {
const prefix = isOrdered ? `${i + 1}. ` : "- ";
const $li = $(li);
// Recurse into nested lists first (depth-first)
$li.children("ol, ul").each((_, nested) => {
const nestedMarkdown = listToMarkdown($, nested, depth + 1);
$(nested).replaceWith($(`${nestedMarkdown}`));
});
// Now get full text (nested lists are replaced with containing markdown)
const itemText = $li.text().trim().replace(/\s+/g, " ");
lines.push(`${indent}${prefix}${itemText}`);
});
return lines.join("\n");
}
export function convertListsToMarkdown(
$: cheerio.CheerioAPI,
ordered: string[] = ["ol"],
unordered: string[] = ["ul"],
): void {
const selectors = [...ordered, ...unordered].join(", ");
// Process from outermost — recursion handles depth-first nesting
$(selectors).each((_, elem) => {
// Skip if already processed (parent already handled this)
if ($(elem).hasClass("list-markdown")) return;
const markdown = listToMarkdown($, elem, 0);
$(elem).replaceWith($(`${markdown}`));
});
}
```
Key insight: recursion goes depth-first. When processing an `li`, nested `ol`/`ul` children are first replaced with `` containing their markdown. Then `$li.text()` includes that markdown. Outer `$(selectors).each(...)` skips already-processed elements.
### `convertGrammarToMarkdown` — grammar production text
`addNewlinesAfterBlocks` runs first globally, injecting `\n` after `emu-production` and `emu-rhs` (added to `BLOCK_ELEMENTS`). Then `.text()` on `emu-grammar` naturally preserves production/alternative boundaries.
```typescript
export function convertGrammarToMarkdown(
$: cheerio.CheerioAPI,
selector: string = "emu-grammar",
): void {
$(selector).each((_, grammar) => {
const text = $(grammar).text().trim();
$(grammar).replaceWith(
$(`\`\`\`bnf\n${text}\n\`\`\``),
);
});
}
```
Add `"emu-production"` and `"emu-rhs"` to `BLOCK_ELEMENTS` array.
### `formatForIngestion` — orchestrator
```typescript
export function formatForIngestion(
$: cheerio.CheerioAPI,
config: Partial = {},
): void {
const cfg = { ...DEFAULT_CONFIG, ...config };
// 1. Inject newlines globally (affects li, p, pre, emu-production, emu-rhs, etc.)
addNewlinesAfterBlocks($);
// 2. Inline leaves (links first — must be before lists/code destroy DOM)
convertLinksToMarkdown($, cfg.links.join(", "));
// 3. Inline code
convertInlineCodeToMarkdown($, cfg.codeBlocks.inline);
// 4. Block leaves (fenced code)
convertBlockCodeToMarkdown($, cfg.codeBlocks.block);
// 5. Structural parents (grammar, lists, tables)
convertGrammarToMarkdown($, cfg.codeBlocks.grammar.join(", "));
convertListsToMarkdown($, cfg.lists.ordered, cfg.lists.unordered);
convertTablesToMarkdown($);
}
```
### Modify existing functions
- **`convertTablesToMarkdown($)`**: No changes needed. `.text()` already extracts markdown text from pre-processed `` elements.
### Modify `setup/ingest.ts`
- Replace individual calls to `convertTablesToMarkdown($)` and `addNewlinesAfterBlocks($)` with single call to `formatForIngestion($)`.
- The child `emu-clause` replacement and deep nesting cleanup remain as-is (before `formatForIngestion`).
Updated call order in `buildSpecDocuments()`:
```typescript
// 1. Child clause replacement (existing)
$section.children("emu-clause").each(...);
$section.find("emu-clause").remove();
// 2. All formatting transformations (single call)
formatForIngestion($);
```
## Files to Modify
1. **`setup/utils/formatHTMLForIngestion.ts`** — add config interface, 5 new functions, orchestrating function
2. **`setup/ingest.ts`** — replace individual calls with `formatForIngestion($)` call
3. **New test file: `setup/utils/formatHTMLForIngestion.test.ts`** — unit tests for each transformation
No new dependencies. Cheerio only.
## Verification
1. **Unit tests**: Write tests for each transformation with sample HTML snippets from the spec:
- `emu-xref a` → markdown link
- `var`/`emu-val`/`emu-const` → inline code
- `pre>code` → fenced code
- `emu-grammar` → fenced bnf
- `ol`/`ul` with nesting → markdown lists
- `table` → markdown table (with links inside cells preserved)
- Full pipeline via `formatForIngestion`
2. **Manual verification**: Run `bun run setup/ingest.ts` and inspect output chunks for correct markdown syntax
3. **Run lint/typecheck**: `bun run lint` and `bun run typecheck` (if available)