opencode-multi-model/.opencode/plans/1773342290743-silent-harbor.md

model

model
gemini-3.1-pro-preview

Plan: Comprehensive Tests for multi-model Tool

1. Overview

The goal is to provide complete test coverage for @.opencode/tools/multi-model.ts by updating the @.opencode/multi-model.test.ts file. Since internal helper functions (like shellQuote, createWindowPlans, etc.) are not exported, we will test all logic through the execute method by mocking shell commands (Bun.$).

A key requirement is to log all mocks that were actually called and verify that the sequence matches expectations.

2. Approach

1. Enhance the Mock Infrastructure:

   • Update setupMockBun$ in multi-model.test.ts to push each executed command to an executedCommands array.
   • Clear executedCommands in beforeEach.
   • For every test case, use expect(executedCommands).toEqual([...]) or expect(executedCommands).toStrictEqual([...]) to assert the exact sequence of shell commands executed.

2. Define Test Cases:

   A. Export & Test Helper Functions

   To ensure robust unit coverage, we will export the internal helper functions from multi-model.ts without cluttering the main export.

   • In multi-model.ts, append an export for __testing_helpers containing functions like:
     • shellQuote
     • normalizeModels
     • findDuplicates
     • createWindowBaseName
     • createWindowPlans
     • levenshtein
     • suggestModels
     • formatInvalidModelError
     • sanitizeName
   • In multi-model.test.ts, import __testing_helpers from ./tools/multi-model.
   • Add comprehensive unit test suites (describe("__testing_helpers", ...)) for each of these functions covering edge cases (e.g. empty inputs, special characters, fuzzy match exact bounds, complex naming collisions).

   B. Validation & Pre-flight Checks (Early Returns)

   These tests verify that the tool bails out early and doesn't execute unnecessary commands.

   • Empty session name: Fails validation. executedCommands should be [].
   • Empty models array: Fails validation. executedCommands should be [].
   • Duplicate models: Fails validation. executedCommands should be [].
   • Not a git repo: Mocks git rev-parse --is-inside-work-tree to fail.
   • Missing tmux: Mocks command -v tmux to fail.
   • Missing opencode: Mocks command -v opencode to fail.
   • opencode models failure: Mocks opencode models to fail.
   • Invalid model IDs (Fuzzy Matching):
     • Mocks opencode models to return a specific list.
     • Test with an invalid model that is close to a valid one (verifies "Did you mean?" suggestions).
     • Test with multiple invalid models.
   • Tmux session already exists: Mocks tmux has-session -t <session> to succeed (return exit code 0).

   B. First Model Failures (isFirst === true)

   The first model sets up the primary tmux session. If it fails, it returns a string immediately instead of collecting errors in failedModels.

   • Worktree path exists: Mock fs.existsSync to return true. Ensure it fails and executedCommands stops after checking tmux session.
   • Git branch exists: Mock git show-ref --verify --quiet refs/heads/... to return exit code 0.
   • Git worktree add fails: Mock git worktree add -b ... to fail.
   • Tmux new-session fails: Mock tmux new-session ... to fail. Verify that undoWorktree (git worktree remove and git branch -D) is called and appended to executedCommands.
   • Tmux send-keys fails: Mock tmux send-keys ... to fail. Verify undoWorktree commands are called.

   C. Subsequent Model Failures (isFirst === false)

   Failures on subsequent models do not abort the process; they append to failedModels.

   • Worktree path exists: Mock fs.existsSync to return true only for the second model's path. Verify first model succeeds, second model is skipped, and result output indicates partial success/failure.
   • Git branch exists: Mock git show-ref to succeed for the second model. Verify it continues and records the failure.
   • Git worktree add fails: Mock git worktree add to fail for the second model.
   • Tmux new-window fails: Mock tmux new-window to fail. Verify undoWorktree commands are executed for the second model's branch/path.
   • Tmux send-keys fails: Mock tmux send-keys to fail for the second model. Verify undoWorktree commands are executed.

   D. Success Cases & Complex Window Names

   • Single model success: Standard happy path. Verify exact command sequence.
   • Multiple models success: Verify tmux new-session is called for the first, and tmux new-window for the rest.
   • Sanitization: Pass a session name with spaces and special chars. Verify the safe name is used in git branch and worktree path commands.
   • Model name collision: Pass models like openai/gpt-4o and anthropic/gpt-4o. Verify the window names in the commands are gpt-4o and gpt-4o-2 respectively.
   • Long model name: Pass a model name that evaluates to > 24 chars for the base window name. Verify truncation in the executed tmux commands.

3. Refactor Existing Unit Tests:

   • Update the "unit tests" at the bottom of the existing file to correctly test __testing_helpers.shellQuote and __testing_helpers.normalizeModels instead of redefining the functions. Expand these unit tests to cover all edge cases mapped out in section A.

3. Implementation Details

We will modify setupMockBun$ in .opencode/multi-model.test.ts:

let executedCommands: string[] = [];

function setupMockBun$() {
  const mockFn = mock((strings: TemplateStringsArray, ...values: any[]) => {
    // Interleave strings and values to reconstruct the command
    let commandSignature = strings[0];
    for (let i = 0; i < values.length; i++) {
      commandSignature += String(values[i]) + strings[i + 1];
    }
    commandSignature = commandSignature.trim().replace(/\s+/g, ' ');

    executedCommands.push(commandSignature);

    // ... (rest of the existing mock logic to return based on mockCommandResponses)
  });
  // ...
}

In beforeEach:

executedCommands = [];
// ... setup default mockCommandResponses for happy path up to tmux has-session ...

Example test case asserting on executedCommands:

test("fails if tmux new-session fails for the first model and triggers undo", async () => {
  mockCommandResponses["tmux new-session -d -s test-session -n gpt-4o -c /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o"] = { ok: false, stdout: "", stderr: "tmux error" };
  
  // mock undo command success
  mockCommandResponses["git worktree remove -f /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o"] = { ok: true, stdout: "", stderr: "" };
  mockCommandResponses["git branch -D opencode/test-session/gpt-4o"] = { ok: true, stdout: "", stderr: "" };

  const result = await multiModelTool.execute(
    { sessionName: "test-session", models: ["openai/gpt-4o"] },
    { metadata: mock() } as any
  );

  expect(result).toContain("Failed to create tmux session");
  
  // Verify exact sequence
  expect(executedCommands).toEqual([
    "git rev-parse --is-inside-work-tree",
    "command -v tmux",
    "command -v opencode",
    "opencode models",
    "tmux has-session -t test-session",
    "git show-ref --verify --quiet refs/heads/opencode/test-session/gpt-4o",
    "git worktree add -b opencode/test-session/gpt-4o /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o",
    "tmux new-session -d -s test-session -n gpt-4o -c /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o",
    // Because it failed, it must run cleanup
    "git worktree remove -f /mock/home/.local/share/opencode/multi-model/test-session/gpt-4o",
    "git branch -D opencode/test-session/gpt-4o"
  ]);
});

4. Next Steps

Once this plan is approved, I will implement all the described test cases in @.opencode/multi-model.test.ts, ensuring that executedCommands accurately captures and verifies every step of the orchestration logic.