ask262/.opencode/plans/1774872666261-parent-breakdown-fix.md

Fix: Parent Section Breakdown Awareness + Sequential Breakdown

Model: fireworks-ai/accounts/fireworks/routers/kimi-k2p5-turbo
Date: 2026-03-31
Parent Plan: 1774872124705-glowing-river.md

Summary

This plan addresses two issues in the spec ingestion process:

1. Parent Awareness: Parent sections now contain references to their subsections
2. Sequential + Recursive Breakdown: Large sections are broken down by trying tags sequentially, and extracted subsections are recursively checked and broken down if still large

Key Features:

• Always extract emu-clause with titles first
• Breakdown tags tried in order: emu-table → emu-grammar → td → p
• Continue to next tag only if previous didn't reduce size enough
• Recursive check: Each extracted subsection is also checked and can be further broken down
• Hierarchical IDs: Show full path like sec-if-statement-emu-table-1-td-2
• Max depth: 3 levels prevents excessive nesting
• Metadata tracking: subsections array lists children (breakdown type derived from IDs)
• Parent references: Agent can discover and fetch the full hierarchy

Files to Modify

File	Changes
setup/ingest.ts	Add parent chunk with subsection awareness + recursive breakdown

Implementation Details

Breakdown Strategy

Phase 1: Extract emu-clause with titles

• Parse all emu-clause elements
• Extract id, title (from h1), and content
• Always keep these as primary structure

Phase 2: Recursive sequential breakdown For each section (emu-clause or subsection) > 5000 chars:

1. Try emu-table - Extract tables as subsections
2. If still large, try emu-grammar - Extract grammar productions
3. If still large, try td - Break table cells
4. If still large, try p - Break paragraphs

Phase 3: Recursive check on extracted subsections

• Each extracted subsection is ALSO checked for size
• If still > 5000 chars, recursively apply breakdown starting from the NEXT tag
• Example: sec-if-statement-emu-table-1 is 8000 chars → try emu-grammar, then td, then p

Stop condition: All chunks (at any level) are < threshold

Configuration

const LARGE_DOC_THRESHOLD = 5000;
const BREAKDOWN_TAGS = ["emu-table", "emu-grammar", "td", "p"] as const;
const MAX_RECURSION_DEPTH = 3;

Breakdown Function

interface BreakdownResult {
  parentDoc: {
    content: string;
    tagUsed: string | null;
    subsections: string[];
  };
  subsectionDocs: Document[];
}

function breakDownSection(
  html: string,
  baseId: string,
  baseTitle: string,
  sourceFile: string,
  parentId: string | null,
  depth: number = 0
): BreakdownResult {
  const $ = cheerio.load(`<div>${html}</div>`);
  const $section = $("div").first();
  
  const fullText = $section.text().trim();
  
  // Base case: small enough or max depth
  if (fullText.length <= LARGE_DOC_THRESHOLD || depth >= MAX_RECURSION_DEPTH) {
    return {
      parentDoc: {
        content: html,
        tagUsed: null,
        subsections: []
      },
      subsectionDocs: []
    };
  }
  
  // Try each breakdown tag sequentially
  const subsectionIds: string[] = [];
  const subsectionDocs: Document[] = [];
  let remainingHtml = html;
  let tagUsed: string | null = null;
  
  for (let i = 0; i < BREAKDOWN_TAGS.length; i++) {
    const tagName = BREAKDOWN_TAGS[i];
    const $temp = cheerio.load(`<div>${remainingHtml}</div>`);
    const $tempSection = $("div").first();
    
    // Check if this tag exists
    if ($tempSection.find(tagName).length === 0) {
      continue;
    }
    
    // Extract elements of this tag
    let counter = 1;
    $tempSection.find(tagName).each((_, elem) => {
      const elemHtml = $(elem).html() || "";
      const elemText = $(elem).text().trim();
      
      if (elemText) {
        const subId = `${baseId}-${tagName}-${counter}`;
        subsectionIds.push(subId);
        
        // Recursively check if this subsection needs further breakdown
        // Start from next tag in sequence (i+1)
        const subResult = breakDownSection(
          elemHtml,
          subId,
          `${baseTitle} [${tagName}]`,
          sourceFile,
          baseId,
          depth + 1
        );
        
        // If subsection was broken down further
        if (subResult.subsectionDocs.length > 0) {
          subsectionDocs.push(...subResult.subsectionDocs);
          
          // Also add the subsection's parent document
          if (subResult.parentDoc.subsections.length > 0) {
            subsectionDocs.push(new Document({
              pageContent: [
                `[Section ${subId}: ${baseTitle} [${tagName}]]`,
                "(Section content below - subsections marked inline)",
                "",
                "---",
                "",
                cheerio.load(subResult.parentDoc.content).text().trim()
              ].join("\n"),
              metadata: {
                source: sourceFile,
                sectionid: subId,
                sectiontitle: `${baseTitle} [${tagName}]`,
                type: "specification",
                parentsectionid: baseId,
                subsections: subResult.parentDoc.subsections,
              },
            }));
          }
        } else {
          // Subsection is small enough, create leaf document
          subsectionDocs.push(new Document({
            pageContent: elemText,
            metadata: {
              source: sourceFile,
              sectionid: subId,
              sectiontitle: `${baseTitle} [${tagName}]`,
              type: "specification",
              parentsectionid: baseId,
              subsections: [],
            },
          }));
        }
        
        counter++;
        // Extract title from element if available
        let elemTitle = "";
        if (tagName === "emu-table") {
          elemTitle = $(elem).find("caption").first().text().trim();
        } else if (tagName === "emu-clause") {
          elemTitle = $(elem).find("h1").first().text().trim();
        }
        
        // Replace with marker line in parent content
        const markerText = elemTitle 
          ? `[Subsection available: title "${elemTitle}" at sectionid: \`${subId}\`]`
          : `[Subsection available at sectionid: \`${subId}\`]`;
        $(elem).replaceWith(`<p>${markerText}</p>`);
      }
    });
    
    // Check if breakdown was effective
    remainingHtml = $tempSection.html() || "";
    const remainingText = $tempSection.text().trim();
    
    if (subsectionIds.length > 0) {
      tagUsed = tagName;
      
      // If remaining is small enough, stop here
      if (remainingText.length <= LARGE_DOC_THRESHOLD) {
        break;
      }
    }
  }
  
  return {
    parentDoc: {
      content: remainingHtml,
      tagUsed,
      subsections: subsectionIds
    },
    subsectionDocs
  };
}

Document Creation with Subsections

async function ingestSpec(): Promise<Document[]> {
  const htmlFiles = await glob(path.join(SPEC_DIR, "*.html"));
  const documents: Document[] = [];

  for (const file of htmlFiles) {
    const content = fs.readFileSync(file, "utf-8");
    const $ = cheerio.load(content);

    $("emu-clause").each((_i, elem) => {
      const id = $(elem).attr("id");
      const title = $(elem).find("h1").first().text().trim();
      const html = $(elem)
        .clone()
        .children("emu-clause")
        .remove()
        .end()
        .html() || "";

      if (!id || !title || !html.trim()) {
        return;
      }

      // Apply recursive breakdown
      const result = breakDownSection(html, id, title, file, null, 0);
      
      // Create parent document
      if (result.parentDoc.subsections.length > 0) {
        const parentContent = [
          `[Section ${id}: ${title}]`,
          "(Section content below - subsections marked inline)",
          "",
          "---",
          "",
          cheerio.load(result.parentDoc.content).text().trim()
        ].join("\n");
        
        documents.push(new Document({
          pageContent: parentContent,
          metadata: {
            source: file,
            sectionid: id,
            sectiontitle: title,
            type: "specification",
            parentsectionid: null,
            subsections: result.parentDoc.subsections,
          },
        }));
        
        // Add all subsection documents (including recursively broken ones)
        documents.push(...result.subsectionDocs);
      } else {
        // No breakdown needed - add as-is
        documents.push(new Document({
          pageContent: cheerio.load(html).text().trim(),
          metadata: {
            source: file,
            sectionid: id,
            sectiontitle: title,
            type: "specification",
            parentsectionid: null,
            subsections: [],
          },
        }));
      }
    });
  }
  return documents;
}

Hierarchical Structure

Recursive breakdown creates nested hierarchy:

sec-if-statement (parent)
├── sec-if-statement-emu-table-1 (parent)
│   ├── sec-if-statement-emu-table-1-td-1
│   ├── sec-if-statement-emu-table-1-td-2
│   └── ...
├── sec-if-statement-emu-table-2 (leaf)
├── sec-if-statement-emu-grammar-1 (leaf)
└── ...

Breakdown type derivation: From subsection ID pattern:

• sec-if-statement-emu-table-1 → broke down by emu-table
• sec-if-statement-emu-table-1-td-2 → table-1 broke down by td

Querying Strategy

Get all chunks from a section (recursive):

async function getAllChunks(sectionId: string): Promise<Document[]> {
  const allDocs: Document[] = [];
  const queue: string[] = [sectionId];
  const visited = new Set<string>();
  
  while (queue.length > 0) {
    const currentId = queue.shift()!;
    if (visited.has(currentId)) continue;
    visited.add(currentId);
    
    const results = await table
      .query()
      .where(`sectionid = '${currentId}'`)
      .limit(100)
      .toArray();
    
    for (const result of results) {
      allDocs.push(result);
      
      // If has subsections, add them to queue
      if (result.subsections && result.subsections.length > 0) {
        queue.push(...result.subsections);
      }
    }
  }
  
  return allDocs;
}

Agent Usage

Parent Discovery

When the agent retrieves a parent chunk (has subsections array with items), it will see inline markers where content was extracted:

[Section sec-if-statement: If Statement]
(Section content below - subsections marked inline)

---

The if statement evaluates a condition...

[Subsection available: title "Static Semantics: Early Errors" at sectionid: `sec-if-statement-emu-table-1`]

The result of the evaluation determines...

[Subsection available: title "IfStatement" at sectionid: `sec-if-statement-emu-grammar-1`]

Further text continues...

Deep Breakdown Example

When a subsection (like a large table) is also broken down:

Parent chunk:

[Section sec-if-statement: If Statement]
(Section content below - subsections marked inline)

---

The if statement evaluates a condition...

[Subsection available: title "Static Semantics: Early Errors" at sectionid: `sec-if-statement-emu-table-1`]

[Subsection available: title "IfStatement" at sectionid: `sec-if-statement-emu-grammar-1`]

The result of the evaluation determines...

Subsection parent (table-1 broken down further by td):

[Section sec-if-statement-emu-table-1: If Statement [emu-table]]
(Section content below - subsections marked inline)

---

Table header row...

[Subsection available at sectionid: `sec-if-statement-emu-table-1-td-1`]

[Subsection available at sectionid: `sec-if-statement-emu-table-1-td-2`]

Table footer...

**Leaf chunk (table cell):**

[Section sec-if-statement-emu-table-1-td-1: If Statement [emu-table] [td]] (Actual table cell content here)

### Agent Strategy

1. Retrieve emu-clause by semantic search
2. Check if `subsections?.length > 0` to detect parent
3. Derive breakdown type from subsection IDs (e.g., `*-emu-table-*` means table breakdown)
4. Recursively check fetched subsections - they may also have subsections!
5. Continue until reaching leaf nodes (no subsections)
6. Combine all retrieved chunks for complete answer

## Changes to agent.ts

### Enhanced fetch_section_chunks Tool

```typescript
const sectionRetrieverTool = new DynamicTool({
  name: "fetch_section_chunks",
  description: "Retrieves all text chunks from a specific specification section by sectionid. " +
               "Supports recursive fetching - if a section has subsections, it will fetch all descendants. " +
               "Use this to get complete content when you see 'Subsection available' references in parent chunks.",
  func: async (sectionId) => {
    const allDocs: string[] = [];
    const queue: string[] = [sectionId];
    const visited = new Set<string>();
    
    while (queue.length > 0) {
      const currentId = queue.shift()!;
      if (visited.has(currentId)) continue;
      visited.add(currentId);
      
      const results = await table
        .query()
        .where(`sectionid = '${currentId}'`)
        .limit(100)
        .toArray();
      
      for (const result of results) {
        allDocs.push(result.text || "");
        
        // Add subsections to queue for recursive fetching
        if (result.subsections && Array.isArray(result.subsections)) {
          queue.push(...result.subsections);
        }
      }
    }
    
    return allDocs.join("\n\n---\n\n");
  }
});

New Check for Breakdown Tool (Optional)

const checkSubsectionsTool = new DynamicTool({
  name: "check_subsections",
  description: "Checks if a section has subsections and returns their IDs. " +
               "Use when you need to selectively fetch specific subsection types (tables, grammar, prose).",
  func: async (sectionId) => {
    const result = await table
      .query()
      .where(`sectionid = '${sectionId}'`)
      .limit(1)
      .toArray();
    
    if (result.length === 0) {
      return `No section found with id: ${sectionId}`;
    }
    
    const doc = result[0];
    if (!doc.subsections || doc.subsections.length === 0) {
      return `Section ${sectionId} has no subsections.`;
    }
    
    return [
      `Section ${sectionId} has ${doc.subsections.length} subsections:`,
      ...doc.subsections.map((id: string) => {
        const match = id.match(/-(table|grammar|prose)/);
        const type = match ? match[1] : 'subsection';
        return `  - ${type}: ${id}`;
      })
    ].join("\n");
  }
});

Testing Checklist

Basic Functionality

• Run bun run setup/ingest.ts completes without errors
• Verify parent chunks have subsections array with children
• Verify leaf chunks have empty/null subsections
• Verify subsection references are in parent chunk content
• Verify subsection IDs show correct breakdown path (e.g., section-emu-table-1)

Sequential Breakdown

• Find a section with >5000 chars and tables, verify emu-table breakdown happens
• Find a section with large grammar block, verify emu-grammar breakdown happens
• Find a section where emu-table leaves large remainder, verify emu-grammar also extracted
• Derive breakdown type from subsection IDs (first tag in ID path)

Recursive Breakdown

• Find a large table (section-emu-table-1 > 5000 chars), verify it gets further broken down
• Check that nested breakdown uses next tags in sequence (emu-grammar, td, p)
• Verify nested IDs: sec-if-statement-emu-table-1-td-2 (table broken down by td)
• Test max depth limit (3): deeply nested sections stop at depth 3
• Verify all subsections have parentsectionid pointing to their immediate parent

Query Testing

• Query for parent section, verify content includes subsection lines
• Query for subsection by ID (e.g., sec-if-statement-emu-table-1)
• Test fetch_section_chunks with parent ID returns all subsections
• Verify agent can discover and fetch subsections automatically
• Test agent query on large section to verify subsection discovery works

Indexes

• Verify scalar indexes are created for: sectionid, type
• Test WHERE clause queries on indexed columns

Limits & Edge Cases

Sequential Breakdown Order

Tags are tried in strict order:

1. emu-table - Best semantic unit for spec tables
2. emu-grammar - Grammar productions are natural boundaries
3. td - Table cells (only if tables couldn't reduce enough)
4. p - Paragraphs (last resort, breaks prose)

Rationale: Structure-aware breakdown preserves semantic meaning better than arbitrary text splitting.

Recursive Breakdown Logic

• Applied to every extracted subsection: Each extracted chunk is also checked for size
• Sequential tags continue: If sec-table-1 is large, try next tags (emu-grammar, td, p)
• Depth tracking: depth field shows nesting level (0 = emu-clause, max 3)
• ID nesting: IDs reflect full path: section-table-1-td-2 means table 1 was broken down by td

Content Size Threshold

• Default: 5000 characters (configurable via LARGE_DOC_THRESHOLD)
• Applied at every level: Parent and all subsections checked against threshold
• Max depth safety: Even if still large at depth 3, no further breakdown (prevents infinite recursion)

Hierarchical Structure

• Nested IDs: IDs show full breakdown path like sec-if-statement-emu-table-1-td-2
• Multi-level: Subsections can be parents with their own children
• Parent tracking: parentsectionid always points to immediate parent (could be subsection or emu-clause)

Large Remainders at Max Depth

If at depth 3 content is still > threshold:

• Keep it as-is (chunk may exceed threshold)
• Add warning in content: "(Content exceeds threshold - max depth reached)"
• This is rare - depth 3 with td/p breakdowns should handle most cases

Migration Strategy

1. Clean slate: Delete existing storage/ directory
2. Re-ingest: Run bun run setup/ingest.ts
3. Verify structure: Check that large sections have hierarchical breakdowns
4. Test breakdown types: Look for examples of each breakdown type by ID pattern:

   // Query to find emu-table breakdowns
   const tableBreakdowns = await table
     .query()
     .where(`sectionid LIKE '%-emu-table-%'`)
     .limit(10)
     .toArray();
   console.log(tableBreakdowns.map(s => s.sectionid));
   

5. Test recursive breakdown: Find deeply nested examples:

   // Query for nested breakdowns (table cells broken down)
   const nested = await table
     .query()
     .where(`sectionid LIKE '%-td-%'`)
     .limit(10)
     .toArray();
   console.log(nested.map(s => ({id: s.sectionid, parent: s.parentsectionid})));
   

6. Test agent: Run queries on large sections like "sec-globaldeclarationinstantiation"
7. Verify hierarchy: Confirm parent-child chain is correct (e.g., td → table → section)