Guides

LLM API testing and evals, without the hosted stack.

Short, practical pages on evaluating models on macOS, talking to provider APIs locally, and when a desktop workbench beats Promptfoo or Postman for day-to-day LLM work.

6 guides

• An LLM evaluation tool that runs on your Mac Run the same prompt across providers on your Mac, compare outputs, and score them with an LLM judge — no hosted eval stack required. Updated August 6, 2026
• A local LLM API client for macOS Compose, stream, and inspect LLM API calls on your Mac with keys that never leave through a Roshi backend. Updated August 6, 2026
• Roshi vs Promptfoo Roshi is a local macOS workbench for exploratory evals; Promptfoo is a CI-oriented framework for large, repeatable prompt suites. Updated August 6, 2026
• Roshi vs Postman for LLM APIs Postman remains the right general HTTP client; Roshi is built for streaming chat APIs, multi-model runs, and local LLM evals on macOS. Updated August 6, 2026
• Compare OpenAI, Claude, and Gemini on one prompt Fair bake-offs need one frozen prompt, parallel runs, and metrics beside the prose — not three browser tabs started minutes apart. Updated August 6, 2026
• Provider API testing and LLM-as-a-judge First prove the provider request path, then grade candidate outputs with a judge model and an explicit rubric — all on your Mac. Updated August 6, 2026