Comparison
Roshi vs Promptfoo
When to use Roshi’s local macOS workbench versus Promptfoo’s YAML-driven eval framework — exploratory model comparison on your machine versus regression suites in CI.
published
Promptfoo and Roshi both help you evaluate LLM behavior. They optimize for different moments in the lifecycle.
Quick contrast
| Roshi | Promptfoo | |
|---|---|---|
| Form factor | Local macOS app | CLI / config-as-code framework |
| Primary job | Compose, compare, inspect, judge interactively | Declare cases, assert, run headlessly |
| Best when | Exploring prompts and vendors on your machine | Guarding regressions in CI |
| Config style | UI + local saved runs | YAML/JS config in the repo |
| Transport | Direct from your Mac to providers | Direct from the runner to providers |
| Team sharing | Export / describe findings | Commit the suite |
Where Promptfoo wins
- Scale. Hundreds of fixtures, matrixed providers, and assertion libraries.
- CI native. Fail the build when a prompt regresses; no GUI required.
- Reviewable config. Diffs in Git beat “I clicked around yesterday.”
- Ecosystem. Red-team plugins, providers, and graders aimed at automation.
If your question is “did this PR make the FAQ bot worse?”, Promptfoo (or a similar framework) is the right class of tool.
Where Roshi wins
- Exploration speed. Tweaking a multi-turn chat with images and custom headers is faster in a dedicated composer than editing YAML between runs.
- Visibility. Streamed output, raw bodies, response headers, and latency sit next to the comparison — useful when the bug is protocol-shaped, not assertion-shaped.
- Local-first UX. No need to bootstrap a Node project to answer “is Gemini good enough for this support tone?”
- Product-shaped evals. Parallel runners + optional LLM judge from a request you already validated.
If your question is “which model should we put into the suite?”, a desktop workbench is usually faster.
A sane combined workflow
- Explore in Roshi: lock a prompt, compare models, maybe judge with a rubric (LLM-as-a-judge).
- Write down the winners and the failure modes you care about.
- Encode those as Promptfoo cases once they stop changing every hour.
- Keep Roshi around for the next exploratory spike; keep Promptfoo for the gate.
Bottom line
Use Promptfoo when evaluation must be declarative, shared, and automated. Use Roshi when evaluation must be interactive, inspectable, and local on macOS. They are complementary, not rivals for the same checkbox.
FAQ
Is Roshi a Promptfoo replacement?
No. Roshi targets interactive, local evaluation and API debugging on macOS. Promptfoo targets declarative suites you run headlessly and wire into CI. Many teams use both at different stages.
Can Promptfoo do LLM-as-a-judge?
Yes. Promptfoo has first-class assertions and model-graded scoring aimed at automated pipelines. Roshi’s judge flow is interactive inside a desktop session with a rubric you apply to a live comparison run.
Which should I start with?
Start with a desktop workbench when you are still choosing models and shaping prompts. Move stable cases into Promptfoo (or similar) when you need every pull request to re-run the same suite.