opencode-plugin-dsml-fix/docs/PATTERNS.md

# DSML Tool-Call Pattern Catalogue

Single source of truth for what the parser **accepts**, what it **repairs**, what it
**rejects**, and which test proves each row. Every regex in `src/grammar.ts` must trace
to a row here. Add a row and a test before adding a pattern.

> Writing rule for this file: never paste a live DSML token. The model may emit it and
> our own harness (or the provider's greedy extractor) will turn this document into a
> phantom tool call. Represent tokens with the placeholder `‹DSML›` and show the
> literal bytes as `U+FF5C` escapes. The parser tests are the place for real bytes.

## 0. Notation

- `BAR` = one full-width bar `U+FF5C` (`｜`) **or** one ASCII pipe `|`.
- `DSML` literal word.
- Canonical marker: `< BAR DSML BAR >` — e.g. `<‹DSML›tool_calls>`.
- Doubled marker (observed): `< BAR BAR DSML BAR BAR >`.
- In real bytes: `\uFF5C` repeated.

## 1. Canonical grammar

```
< MARKER tool_calls>
  < MARKER invoke name="TOOL">
    < MARKER parameter name="KEY" string="true">VALUE</ MARKER parameter>
    < MARKER parameter name="KEY" string="false">JSON</ MARKER parameter>
  </ MARKER invoke>
</ MARKER tool_calls>
```

- Outer block names: `tool_calls` (V4), `function_calls` (V3.2), and the mangled `calls`.
- Single or doubled bars; optional whitespace around the marker.
- `string="true"` → raw string value. `string="false"` → JSON value. Attribute absent →
  raw string (default). Quotes may be `"` or `'`.
- Invoke/param names may be quoted with `"`/`'` or be a bare token.

## 2. Leak / degradation variants (the real-world enemies)

These are the shapes observed in the wild, in the order a stream degrades. Source tags:
`[DB]` = captured in the user's OpenCode session DB, `[vLLM]` = vLLM PR #54686,
`[OpenClaw]` = openclaw doubled-bar issue, `[Pi]` = pi-dsml grammar, `[CS]` = Cherry Studio.

| # | Variant | Shape | Handle by | Test |
|---|---------|-------|-----------|------|
| V1 | Canonical single-bar | `< BAR DSML BAR tool_calls>` complete | parse normally | `v1-canonical` |
| V2 | Doubled full-width bars | `< BAR BAR DSML BAR BAR tool_calls>` | marker accepts 1–2 bars | `v2-doubled-bars` |
| V3 | ASCII pipes | `< | DSML | tool_calls>` | marker accepts `|` | `v3-ascii-pipes` |
| V4 | Space inside marker | `< BAR DSML BAR  tool_calls>` / `calls` | trailing `\s*` + `calls` alt | `v4-marker-space` |
| V5 | Truncated block tag | `< BAR DSML BAR calls>` | `calls` alternative for `tool_calls` | `v5-truncated-tag` |
| V6 | Spaces around invoke name | `< BAR DSML BAR invoke name = "x">` | `\s+` around `name=` | `v6-name-spacing` |
| V7 | `tool_call` (singular) | `< BAR DSML BAR tool_call>` | block tag alt | `v7-tool-call-singular` |
| V8 | `function_calls` (V3.2) | `< BAR DSML BAR function_calls>` | block tag alt | `v8-function-calls` |
| V9 | **Missing outer opener** | invoke/parameter appear with no `tool_calls` block | implicit-block rescue | `v9-orphan-invoke` |
| V10 | **Missing invoke opener** | leaked `parameter` only, no `invoke name=` | implicit-invoke rescue | `v10-orphan-parameter` |
| V11 | Mis-spelled closer | `</ BAR DSML BAR >` (bar then `>`), swallowing next param | tolerant scan + drop runaway param | `v11-misspelled-closer` |
| V12 | Runaway invoke name | `invoke name="tool {` (JSON leaked into name) | reject call, strip text | `v12-runaway-name` |
| V13 | Undeclared tool | well-formed, name not in tool set | **keep call**; runtime answers "tool not found" | `v13-undeclared-tool` |
| V14 | Empty invoke name | `name=""` | reject (never trap buffer) | `v14-empty-name` |
| V15 | Unclosed block at end | opener, no closer, stream ends | flush as text, recover complete calls only | `v15-unclosed` |
| V16 | Truncated mid-invoke | invoke has no closing tag | drop partial invoke, keep earlier ones | `v16-truncated-invoke` |
| V17 | Chunk-split opener | `<` `BAR` `DSML` … across deltas | suffix-prefix buffering | `v17-chunk-split` |
| V18 | Text before/after block | prose + block + prose | preserve prose, excise block | `v18-surrounding` |
| V19 | Multiple invokes / blocks | 2+ calls in one or many blocks | parse all | `v19-multiple` |
| V20 | Inside fenced code | grammar shown in ``` fence | **ignore** (no parse, no strip) | `v20-code-fence` |
| V21 | Trailing `‹EOS›` token | `…</…tool_calls>‹EOS›` | strip EOS token | `v21-eos` |
| V22 | HTML entities in values | `&quot;` `&amp;` `&lt;` | decode entities | `v22-entities` |
| V23 | Multi-line / heredoc value | value contains newlines, indentation | preserve verbatim except hugging newline | `v23-multiline-value` |
| V24 | JSON params | `string="false"` | `JSON.parse`, fall back to raw on failure | `v24-json-param` |
| V25 | Raw JSON body (no params) | invoke body is a bare `{...}` | fallback JSON parse | `v25-raw-json-body` |
| V26 | **Wrapped tool** (dominant real shape) | orphan `<parameter name="X">` (marker optional) containing ≥1 complete inner `<MARKER parameter>` | recover call X with inner args (V13 policy: runtime validates) | `v26-*` |

## 3. Reject arms (never emit a call)

The parser is a **strict acceptor**. When in doubt it does not execute; it either strips
noise or leaves text alone. This is deliberate: a false positive executes an arbitrary
tool the model never asked for.

| Reject arm | Rationale | Test |
|-----------|-----------|------|
| No DSML marker anywhere | fast path, zero cost | `reject-no-marker` |
| Text inside a code fence | docs/examples must not execute | `v20-code-fence` |
| Complete call but empty name | unusable, and must not trap buffer | `v14-empty-name` |
| Invoke with no parseable parameter and no JSON body | nothing to run | `reject-empty-args` |
| Native tool calls already present on the message | strip text only, do not double-run | `reject-native-present` |
| `looksLikeDsml` false (no `DSML`/`end-of-sentence`) | skip whole pipeline | `reject-not-dsml` |

## 4. Transition: removal in old transcripts vs. rejection in new streams

The user wants the same engine to serve **two consumers**:

1. **Streaming (new turns, `ctx.aisdk.language`)** — convert a DSML block into a real
   `tool-call` part *before* OpenCode's lowering, and rewrite `finishReason` to
   `tool-calls`. The block must never reach the transcript.
2. **Stored transcript repair (old sessions)** — walk existing messages, excise DSML
   spans, and (for messages that never got a native call) synthesize the missing call.

Both consume the **same `parseDsml(text) -> { calls, ranges }`** function. The only
difference is the sink:

- Stream sink: buffer `text-delta`, emit `tool-input-*` + `tool-call` for each call,
  drop `ranges` from the emitted text, set `finish` to `tool-calls` if any call.
- Repair sink: `stripRangesWithNote` for display (short inline note naming the
  removed tool when known, e.g. `[removed malformed edit call]`), append calls if
  none were native.

Because both share the parser, a variant is "supported" only when both sinks agree, and
the test table covers both.

## 5. Streaming buffer policy

- Hold text ONLY from the first byte that could open a call: a DSML opener, a
  partial-opener suffix (`<`, `<BAR`, `<parameter name="sh`, …), or a wrapped-tool
  candidate (`<parameter name="X">` with or without marker). Everything before it is
  emitted immediately, every chunk — plain prose is never buffered.
- Eager stop: a held candidate that produces no complete parameter-open within
  1024 chars is prose — flush as text and resume scanning fresh. Once a complete
  parameter is open (inside a value), only the 64 KiB cap bounds the hold.
- Transitions: any non-text part (reasoning, native tool parts) flushes held text
  FIRST, preserving transcript order.
- End-of-text guarantee: `text-end` / `finish` / stream flush always resolve the
  buffer — recovered calls plus verbatim remainder — so the buffer always ends
  empty and no text is ever lost.
- Trailing markup noise (`</invoke>` / `</calls>` arriving after an already
  recovered call) is dropped, not leaked.