mirror of
https://github.com/bendtherules/opencode-multi-model.git
synced 2026-08-18 13:42:21 +00:00
182 lines
11 KiB
Markdown
182 lines
11 KiB
Markdown
---
|
|
model: gemini-3.1-pro-preview
|
|
---
|
|
|
|
# Plan: Comprehensive Tests for `multi-model` Tool
|
|
|
|
## 1. Overview
|
|
The goal is to provide complete test coverage for `@.opencode/tools/multi-model.ts` by updating the `@.opencode/multi-model.test.ts` file. Since internal helper functions (like `shellQuote`, `createWindowPlans`, etc.) are not exported, we will test all logic through the `execute` method by mocking shell commands (`Bun.$`).
|
|
|
|
A key requirement is to **log all mocks that were actually called and verify that the sequence matches** expectations.
|
|
|
|
## 2. Approach
|
|
|
|
1. **Enhance the Mock Infrastructure:**
|
|
- Update `setupMockBun$` in `multi-model.test.ts` to push each executed command to an `executedCommands` array.
|
|
- Clear `executedCommands` in `beforeEach`.
|
|
- For every test case, use `expect(executedCommands).toEqual([...])` or `expect(executedCommands).toStrictEqual([...])` to assert the exact sequence of shell commands executed.
|
|
|
|
2. **Define Test Cases:**
|
|
|
|
### A. Export & Test Helper Functions
|
|
To ensure robust unit coverage, we will export the internal helper functions from `multi-model.ts` without cluttering the main export.
|
|
* In `multi-model.ts`, append an export for `__testing_helpers` containing functions.
|
|
* In `multi-model.test.ts`, import `__testing_helpers` from `./tools/multi-model`.
|
|
* Add comprehensive unit test suites (`describe("__testing_helpers", ...)`) for each function covering:
|
|
* `shellQuote`:
|
|
* Normal string without special characters.
|
|
* String with spaces.
|
|
* String with single quotes (verifying `'"'"'` replacement, e.g., `test'value`).
|
|
* String with multiple single quotes.
|
|
* Empty string.
|
|
* `normalizeModels`:
|
|
* Empty array.
|
|
* `undefined` input.
|
|
* Array with whitespace-only strings.
|
|
* Array with mixed valid models, padded models, and empty strings (verifying trimming and filtering).
|
|
* `findDuplicates`:
|
|
* Array with no duplicates.
|
|
* Array with one duplicate pair.
|
|
* Array with multiple different duplicates.
|
|
* Array with a single value repeated more than twice (should only return it once).
|
|
* Verifying first-repeated-occurrence order.
|
|
* `createWindowBaseName`:
|
|
* Normal model name without slashes.
|
|
* Model with slashes (verifying extraction of the last part, e.g., `vendor/namespace/model`).
|
|
* Model exceeding `WINDOW_NAME_LIMIT` (24 chars) (verifying truncation).
|
|
* Model with special characters (verifying lowercase, hyphen replacement, and trimming of leading/trailing hyphens).
|
|
* Model made entirely of special characters (fallback to `"model"`).
|
|
* `createWindowPlans`:
|
|
* Single model (verifying original base name).
|
|
* Two models with the same base name (verifying `-2` suffix for the second).
|
|
* Multiple collisions (verifying `-3`, `-4`, etc.).
|
|
* Verifying truncation of the base name to accommodate the suffix without exceeding `WINDOW_NAME_LIMIT` (e.g., base name length 24 + suffix length 2 -> base name truncated to 22).
|
|
* `levenshtein`:
|
|
* Identical strings (distance 0).
|
|
* One substitution, insertion, or deletion (distance 1).
|
|
* Completely different strings.
|
|
* One empty string, both empty strings.
|
|
* `suggestModels`:
|
|
* Exact match (distance 0).
|
|
* Fuzzy match/typo (e.g., `gpt5.4` matching `gpt-5.4`).
|
|
* Fuzzy match with contains boost (e.g., `gpt4o` requesting `gpt-4o` vs `gpt-4o-mini`).
|
|
* Verifying maximum of 3 suggestions returned.
|
|
* Sorting logic (score first, then alphabetical fallback).
|
|
* `formatInvalidModelError`:
|
|
* Single invalid model with available suggestions.
|
|
* Single invalid model without suggestions (fallback message to run `opencode models`).
|
|
* Multiple invalid models (verifying formatting of comma-separated list and suggestions based on the first invalid model).
|
|
* `sanitizeName`:
|
|
* Normal alphanumeric name.
|
|
* Name with spaces and special characters (verifying replacement with hyphens).
|
|
* Name with consecutive hyphens (verifying deduplication).
|
|
* Name with leading/trailing hyphens (verifying trimming).
|
|
|
|
### B. Validation & Pre-flight Checks (Early Returns)
|
|
These tests verify that the tool bails out early and doesn't execute unnecessary commands. Note: Existing tests marked as `[x]` will be updated to also assert on `executedCommands` being `[]`.
|
|
* [x] **Empty session name:** Fails validation. `executedCommands` should be `[]`.
|
|
* [x] **Empty models array:** Fails validation. `executedCommands` should be `[]`.
|
|
* [x] **Duplicate models:** Fails validation. `executedCommands` should be `[]`.
|
|
* [x] **Not a git repo:** Mocks `git rev-parse --is-inside-work-tree` to fail.
|
|
* [x] **Missing `tmux`:** Mocks `command -v tmux` to fail.
|
|
* [x] **Missing `opencode`:** Mocks `command -v opencode` to fail.
|
|
* [ ] **`opencode models` failure:** Mocks `opencode models` to fail.
|
|
* [x] **Invalid model IDs (Fuzzy Matching):**
|
|
* [x] Mocks `opencode models` to return a specific list.
|
|
* [ ] Test with an invalid model that is close to a valid one (verifies "Did you mean?" suggestions).
|
|
* [ ] Test with multiple invalid models.
|
|
* [x] **Tmux session already exists:** Mocks `tmux has-session -t <session>` to succeed (return exit code 0).
|
|
|
|
### C. First Model Failures (isFirst === true)
|
|
The first model sets up the primary tmux session. If it fails, it returns a string immediately instead of collecting errors in `failedModels`.
|
|
* [ ] **Worktree path exists:** Mock `fs.existsSync` to return `true`. Ensure it fails and `executedCommands` stops after checking tmux session.
|
|
* [ ] **Git branch exists:** Mock `git show-ref --verify --quiet refs/heads/...` to return exit code 0.
|
|
* [ ] **Git worktree add fails:** Mock `git worktree add -b ...` to fail.
|
|
* [ ] **Tmux new-session fails:** Mock `tmux new-session ...` to fail. Verify that `undoWorktree` (`git worktree remove` and `git branch -D`) is called and appended to `executedCommands`.
|
|
* [ ] **Tmux send-keys fails:** Mock `tmux send-keys ...` to fail. Verify `undoWorktree` commands are called.
|
|
|
|
### D. Subsequent Model Failures (isFirst === false)
|
|
Failures on subsequent models do not abort the process; they append to `failedModels`.
|
|
* [ ] **Worktree path exists:** Mock `fs.existsSync` to return `true` only for the second model's path. Verify first model succeeds, second model is skipped, and result output indicates partial success/failure.
|
|
* [ ] **Git branch exists:** Mock `git show-ref` to succeed for the second model. Verify it continues and records the failure.
|
|
* [ ] **Git worktree add fails:** Mock `git worktree add` to fail for the second model.
|
|
* [ ] **Tmux new-window fails:** Mock `tmux new-window` to fail. Verify `undoWorktree` commands are executed for the second model's branch/path.
|
|
* [ ] **Tmux send-keys fails:** Mock `tmux send-keys` to fail for the second model. Verify `undoWorktree` commands are executed.
|
|
|
|
### E. Success Cases & Complex Window Names
|
|
Note: Existing tests marked as `[x]` will be updated to assert on the exact `executedCommands` sequence.
|
|
* [x] **Single model success:** Standard happy path. Verify exact command sequence.
|
|
* [x] **Multiple models success:** Verify `tmux new-session` is called for the first, and `tmux new-window` for the rest.
|
|
* [x] **Sanitization:** Pass a session name with spaces and special chars. Verify the safe name is used in git branch and worktree path commands.
|
|
* [ ] **Model name collision:** Pass models like `openai/gpt-4o` and `anthropic/gpt-4o`. Verify the window names in the commands are `gpt-4o` and `gpt-4o-2` respectively.
|
|
* [ ] **Long model name:** Pass a model name that evaluates to > 24 chars for the base window name. Verify truncation in the executed `tmux` commands.
|
|
|
|
3. **Refactor Existing Unit Tests:**
|
|
- Update the "unit tests" at the bottom of the existing file to correctly test `__testing_helpers.shellQuote` and `__testing_helpers.normalizeModels` instead of redefining the functions. Expand these unit tests to cover all edge cases mapped out in section A.
|
|
|
|
## 3. Implementation Details
|
|
|
|
We will modify `setupMockBun$` in `.opencode/multi-model.test.ts`:
|
|
|
|
```typescript
|
|
let executedCommands: string[] = [];
|
|
|
|
function setupMockBun$() {
|
|
const mockFn = mock((strings: TemplateStringsArray, ...values: any[]) => {
|
|
// Interleave strings and values to reconstruct the command
|
|
let commandSignature = strings[0];
|
|
for (let i = 0; i < values.length; i++) {
|
|
commandSignature += String(values[i]) + strings[i + 1];
|
|
}
|
|
commandSignature = commandSignature.trim().replace(/\s+/g, ' ');
|
|
|
|
executedCommands.push(commandSignature);
|
|
|
|
// ... (rest of the existing mock logic to return based on mockCommandResponses)
|
|
});
|
|
// ...
|
|
}
|
|
```
|
|
|
|
In `beforeEach`:
|
|
```typescript
|
|
executedCommands = [];
|
|
// ... setup default mockCommandResponses for happy path up to tmux has-session ...
|
|
```
|
|
|
|
Example test case asserting on `executedCommands`:
|
|
```typescript
|
|
test("fails if tmux new-session fails for the first model and triggers undo", async () => {
|
|
mockCommandResponses["tmux new-session -d -s test-session -n gpt-4o -c /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o"] = { ok: false, stdout: "", stderr: "tmux error" };
|
|
|
|
// mock undo command success
|
|
mockCommandResponses["git worktree remove -f /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o"] = { ok: true, stdout: "", stderr: "" };
|
|
mockCommandResponses["git branch -D opencode/test-session/gpt-4o"] = { ok: true, stdout: "", stderr: "" };
|
|
|
|
const result = await multiModelTool.execute(
|
|
{ sessionName: "test-session", models: ["openai/gpt-4o"] },
|
|
{ metadata: mock() } as any
|
|
);
|
|
|
|
expect(result).toContain("Failed to create tmux session");
|
|
|
|
// Verify exact sequence
|
|
expect(executedCommands).toEqual([
|
|
"git rev-parse --is-inside-work-tree",
|
|
"command -v tmux",
|
|
"command -v opencode",
|
|
"opencode models",
|
|
"tmux has-session -t test-session",
|
|
"git show-ref --verify --quiet refs/heads/opencode/test-session/gpt-4o",
|
|
"git worktree add -b opencode/test-session/gpt-4o /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o",
|
|
"tmux new-session -d -s test-session -n gpt-4o -c /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o",
|
|
// Because it failed, it must run cleanup
|
|
"git worktree remove -f /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o",
|
|
"git branch -D opencode/test-session/gpt-4o"
|
|
]);
|
|
});
|
|
```
|
|
|
|
## 4. Next Steps
|
|
Once this plan is approved, I will implement all the described test cases in `@.opencode/multi-model.test.ts`, ensuring that `executedCommands` accurately captures and verifies every step of the orchestration logic.
|