Compare OpenAI, Claude, and Gemini on one prompt

“Claude feels better” is not a decision you can ship. A multi-model comparison turns that feeling into a recorded run: same prompt, several providers, outputs and metrics you can reopen tomorrow.

Freeze the request first

Before you touch model dropdowns:

1. Lock the messages. System prompt, few-shot turns, and user input should be identical for every runner.
2. Lock sampling knobs. Temperature, max tokens, and other parameters should match unless the experiment is the parameter.
3. Lock the task. “Summarize this PDF,” “emit valid JSON,” and “be a witty chatbot” are different contests.
4. Note the context. Attach the same files/images; do not quietly give one model extra paste.

If the request is still changing, you are still composing — not comparing. Use a local API client until the request is boring.

Run in parallel, not in series

Serial playground tabs introduce clock skew, rate-limit weather, and “I tweaked the prompt after the first answer.” Parallel runners on one frozen request keep the bake-off honest.

Typical matrix for product work:

• OpenAI (e.g. your production GPT class)
• Anthropic Claude (Sonnet / Opus class you actually consider)
• Google Gemini (the tier you would enable)
• Optional OpenRouter route for a dark-horse or open-weight candidate

Read more than the prose

For each runner, check:

• Final text — correctness and style
• Streaming behavior — time-to-first-token vs complete answer
• Tokens — input/output sizes that drive cost
• Errors / refusals / truncated JSON — invisible in a screenshot of the happy path
• Headers / finish reasons — when the API, not the model, is the problem

Score when needed

Human ranking works for three candidates. For ten, or for a rubric legal cares about, add LLM-as-a-judge: pick a judge model, write pass/fail criteria, and store the scores with the run.

Save the evidence

Name the eval, keep it on disk, export JSON/CSV if you need to share. Future-you will not remember which temperature produced the good Claude answer.

Do it in Roshi

Roshi on macOS is built around this workflow: one composed request, multiple provider/model runners, side-by-side review, optional judge, local storage. No Roshi account; keys stay on your machine.