An LLM evaluation tool that runs on your Mac

Most “LLM evaluation platforms” assume you will upload prompts, keys, and outputs to someone else’s cloud. That is fine for large shared suites. It is awkward when you just want to know whether Claude, GPT, or Gemini is better for this request on your Mac, without standing up another SaaS.

A local-first evaluation workflow flips that: providers stay where they already are, your keys never leave the machine through a middleman, and you can inspect the raw response — not only a summary card.

What “LLM eval” usually means in practice

People use the phrase for several different jobs:

1. Side-by-side comparison — same prompt, several models, read the answers.
2. Metric comparison — latency, tokens, cost signals, finish reasons.
3. Judged scoring — another model grades candidates with a rubric (LLM-as-a-judge).
4. Regression suites — hundreds of cases in CI, often YAML-driven.

A macOS workbench is strongest on (1)–(3). CI frameworks remain the right tool for (4) once the suite is boring and repeatable.

A practical Mac-native loop

1. Put providers where you already pay for them.
Add OpenAI, Anthropic, Google Gemini, OpenRouter, or any compatible endpoint. Keys and custom headers live in local app storage, not in a hosted vault you did not ask for.

2. Start from a real request.
Use the conversation you are actually shipping — system prompt, tools-shaped messages, images if needed — not a toy “Hello world.”

3. Fan the same request out.
Run multiple provider/model pairs in parallel so you are not comparing Tuesday’s Claude run to Wednesday’s GPT run under different temperatures.

4. Read outputs and metrics together.
Look at the streamed text, raw body, headers, latency, and token usage before you declare a winner. Pretty formatting can hide refusals, truncated JSON, or silent tool-call failures.

5. Score when vibes are not enough.
Pick a judge model and write a short rubric (“correct JSON schema,” “cites only the provided context,” “no hallucinated API fields”). Save the run so you can reopen it after the next prompt tweak.

That multi-model comparison workflow and LLM-as-a-judge setup are covered in more detail in the related guides.

Why local-first matters for evals

• Keys stay on your machine. Eval platforms that proxy traffic create another place your production prompts and secrets can leak.
• No account tax. You should not need a new login to compare two models you already pay for.
• Inspectability. Streamed chunks, response headers, and exportable code matter when you are debugging provider quirks, not only ranking prose quality.
• Offline-friendly iteration. Drafts, history, and saved eval runs live on disk even when you are not staring at a browser tab.

When to reach for something else

Choose a CI-oriented framework (see Roshi vs Promptfoo) when you need headless runs, shared fixtures across a team, or a gate that fails the build. Keep a desktop client when the question is still “which model should we even put in that suite?”

Try the workflow

Roshi is a local-first macOS app built for this loop: compose a request, run it across providers, compare responses, and optionally score them with an LLM judge. Free from GitHub Releases, or $0.99 on the Mac App Store if you want automatic updates.