opencode-multi-model/.opencode/plans/archive/1773342290743-silent-harbor.md

model

model
gemini-3.1-pro-preview

Plan: Comprehensive Tests for multi-model Tool

1. Overview

The goal is to provide complete test coverage for @.opencode/tools/multi-model.ts by updating the @.opencode/multi-model.test.ts file. Since internal helper functions (like shellQuote, createWindowPlans, etc.) are not exported, we will test all logic through the execute method by mocking shell commands (Bun.$).

A key requirement is to log all mocks that were actually called and verify that the sequence matches expectations.

2. Approach

1. Enhance the Mock Infrastructure:

   • Update setupMockBun$ in multi-model.test.ts to push each executed command to an executedCommands array.
   • Clear executedCommands in beforeEach.
   • For every test case, use expect(executedCommands).toEqual([...]) or expect(executedCommands).toStrictEqual([...]) to assert the exact sequence of shell commands executed.

2. Define Test Cases:

   A. Export & Test Helper Functions

   To ensure robust unit coverage, we will export the internal helper functions from multi-model.ts without cluttering the main export.

   • In multi-model.ts, append an export for __testing_helpers containing functions.
   • In multi-model.test.ts, import __testing_helpers from ./tools/multi-model.
   • Add comprehensive unit test suites (describe("__testing_helpers", ...)) for each function covering:
     • shellQuote:
       • Normal string without special characters.
       • String with spaces.
       • String with single quotes (verifying '"'"' replacement, e.g., test'value).
       • String with multiple single quotes.
       • Empty string.
     • normalizeModels:
       • Empty array.
       • undefined input.
       • Array with whitespace-only strings.
       • Array with mixed valid models, padded models, and empty strings (verifying trimming and filtering).
     • findDuplicates:
       • Array with no duplicates.
       • Array with one duplicate pair.
       • Array with multiple different duplicates.
       • Array with a single value repeated more than twice (should only return it once).
       • Verifying first-repeated-occurrence order.
     • createWindowBaseName:
       • Normal model name without slashes.
       • Model with slashes (verifying extraction of the last part, e.g., vendor/namespace/model).
       • Model exceeding WINDOW_NAME_LIMIT (24 chars) (verifying truncation).
       • Model with special characters (verifying lowercase, hyphen replacement, and trimming of leading/trailing hyphens).
       • Model made entirely of special characters (fallback to "model").
     • createWindowPlans:
       • Single model (verifying original base name).
       • Two models with the same base name (verifying -2 suffix for the second).
       • Multiple collisions (verifying -3, -4, etc.).
       • Verifying truncation of the base name to accommodate the suffix without exceeding WINDOW_NAME_LIMIT (e.g., base name length 24 + suffix length 2 -> base name truncated to 22).
     • levenshtein:
       • Identical strings (distance 0).
       • One substitution, insertion, or deletion (distance 1).
       • Completely different strings.
       • One empty string, both empty strings.
     • suggestModels:
       • Exact match (distance 0).
       • Fuzzy match/typo (e.g., gpt5.4 matching gpt-5.4).
       • Fuzzy match with contains boost (e.g., gpt4o requesting gpt-4o vs gpt-4o-mini).
       • Verifying maximum of 3 suggestions returned.
       • Sorting logic (score first, then alphabetical fallback).
     • formatInvalidModelError:
       • Single invalid model with available suggestions.
       • Single invalid model without suggestions (fallback message to run opencode models).
       • Multiple invalid models (verifying formatting of comma-separated list and suggestions based on the first invalid model).
     • sanitizeName:
       • Normal alphanumeric name.
       • Name with spaces and special characters (verifying replacement with hyphens).
       • Name with consecutive hyphens (verifying deduplication).
       • Name with leading/trailing hyphens (verifying trimming).

   B. Validation & Pre-flight Checks (Early Returns)

   These tests verify that the tool bails out early and doesn't execute unnecessary commands. Note: Existing tests marked as [x] will be updated to also assert on executedCommands being [].

   • Empty session name: Fails validation. executedCommands should be [].
   • Empty models array: Fails validation. executedCommands should be [].
   • Duplicate models: Fails validation. executedCommands should be [].
   • Not a git repo: Mocks git rev-parse --is-inside-work-tree to fail.
   • Missing tmux: Mocks command -v tmux to fail.
   • Missing opencode: Mocks command -v opencode to fail.
   • opencode models failure: Mocks opencode models to fail.
   • Invalid model IDs (Fuzzy Matching):
     • Mocks opencode models to return a specific list.
     • Test with an invalid model that is close to a valid one (verifies "Did you mean?" suggestions).
     • Test with multiple invalid models.
   • Tmux session already exists: Mocks tmux has-session -t <session> to succeed (return exit code 0).

   C. First Model Failures (isFirst === true)

   The first model sets up the primary tmux session. If it fails, it returns a string immediately instead of collecting errors in failedModels.

   • Worktree path exists: Mock fs.existsSync to return true. Ensure it fails and executedCommands stops after checking tmux session.
   • Git branch exists: Mock git show-ref --verify --quiet refs/heads/... to return exit code 0.
   • Git worktree add fails: Mock git worktree add -b ... to fail.
   • Tmux new-session fails: Mock tmux new-session ... to fail. Verify that undoWorktree (git worktree remove and git branch -D) is called and appended to executedCommands.
   • Tmux send-keys fails: Mock tmux send-keys ... to fail. Verify undoWorktree commands are called.

   D. Subsequent Model Failures (isFirst === false)

   Failures on subsequent models do not abort the process; they append to failedModels.

   • Worktree path exists: Mock fs.existsSync to return true only for the second model's path. Verify first model succeeds, second model is skipped, and result output indicates partial success/failure.
   • Git branch exists: Mock git show-ref to succeed for the second model. Verify it continues and records the failure.
   • Git worktree add fails: Mock git worktree add to fail for the second model.
   • Tmux new-window fails: Mock tmux new-window to fail. Verify undoWorktree commands are executed for the second model's branch/path.
   • Tmux send-keys fails: Mock tmux send-keys to fail for the second model. Verify undoWorktree commands are executed.

   E. Success Cases & Complex Window Names

   Note: Existing tests marked as [x] will be updated to assert on the exact executedCommands sequence.

   • Single model success: Standard happy path. Verify exact command sequence.
   • Multiple models success: Verify tmux new-session is called for the first, and tmux new-window for the rest.
   • Sanitization: Pass a session name with spaces and special chars. Verify the safe name is used in git branch and worktree path commands.
   • Model name collision: Pass models like openai/gpt-4o and anthropic/gpt-4o. Verify the window names in the commands are gpt-4o and gpt-4o-2 respectively.
   • Long model name: Pass a model name that evaluates to > 24 chars for the base window name. Verify truncation in the executed tmux commands.

3. Refactor Existing Unit Tests:

   • Update the "unit tests" at the bottom of the existing file to correctly test __testing_helpers.shellQuote and __testing_helpers.normalizeModels instead of redefining the functions. Expand these unit tests to cover all edge cases mapped out in section A.

3. Implementation Details

We will modify setupMockBun$ in .opencode/multi-model.test.ts:

let executedCommands: string[] = [];

function setupMockBun$() {
  const mockFn = mock((strings: TemplateStringsArray, ...values: any[]) => {
    // Interleave strings and values to reconstruct the command
    let commandSignature = strings[0];
    for (let i = 0; i < values.length; i++) {
      commandSignature += String(values[i]) + strings[i + 1];
    }
    commandSignature = commandSignature.trim().replace(/\s+/g, ' ');

    executedCommands.push(commandSignature);

    // ... (rest of the existing mock logic to return based on mockCommandResponses)
  });
  // ...
}

In beforeEach:

executedCommands = [];
// ... setup default mockCommandResponses for happy path up to tmux has-session ...

Example test case asserting on executedCommands:

test("fails if tmux new-session fails for the first model and triggers undo", async () => {
  mockCommandResponses["tmux new-session -d -s test-session -n gpt-4o -c /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o"] = { ok: false, stdout: "", stderr: "tmux error" };
  
  // mock undo command success
  mockCommandResponses["git worktree remove -f /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o"] = { ok: true, stdout: "", stderr: "" };
  mockCommandResponses["git branch -D opencode/test-session/gpt-4o"] = { ok: true, stdout: "", stderr: "" };

  const result = await multiModelTool.execute(
    { sessionName: "test-session", models: ["openai/gpt-4o"] },
    { metadata: mock() } as any
  );

  expect(result).toContain("Failed to create tmux session");
  
  // Verify exact sequence
  expect(executedCommands).toEqual([
    "git rev-parse --is-inside-work-tree",
    "command -v tmux",
    "command -v opencode",
    "opencode models",
    "tmux has-session -t test-session",
    "git show-ref --verify --quiet refs/heads/opencode/test-session/gpt-4o",
    "git worktree add -b opencode/test-session/gpt-4o /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o",
    "tmux new-session -d -s test-session -n gpt-4o -c /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o",
    // Because it failed, it must run cleanup
    "git worktree remove -f /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o",
    "git branch -D opencode/test-session/gpt-4o"
  ]);
});

4. Next Steps

Once this plan is approved, I will implement all the described test cases in @.opencode/multi-model.test.ts, ensuring that executedCommands accurately captures and verifies every step of the orchestration logic.