Eval technique
Provider API testing and LLM-as-a-judge
How to combine direct provider API testing with LLM-as-a-judge scoring — validate the request path first, then grade candidate model outputs with a rubric on your Mac.
published
Two different failures get blamed on “the model.” Sometimes the provider request is wrong — bad header, wrong model id, truncated stream. Sometimes the request is fine and the model answer is weak. Provider API testing catches the first. LLM-as-a-judge helps with the second.
Part 1 — Prove the provider path
Before you trust any comparison:
- Configure the provider explicitly. Base URL, auth, headers, chat endpoint, model list — especially for compatible gateways.
- Send a minimal known-good call. Confirm 2xx, streaming, and a sensible body.
- Inspect the wire. Latency, response headers, and raw chunks reveal proxy issues, region errors, and content-filter responses that a polished UI might soften.
- Escalate to the real prompt. Only then attach your production-like messages and parameters.
A local LLM API client exists for this loop. If the path is broken, no rubric will save you.
Part 2 — Grade candidates with a judge
Once several models answer the same frozen prompt (multi-model comparison):
- Write a short rubric. Example: “Returns valid JSON with keys
title,tags; tags ≤ 5; no markdown fences; refuses if the user asks for medical dosage.” - Pick a judge model. Strong instruction following matters more than brand loyalty.
- Run the judge over each candidate. Store numeric or pass/fail scores next to the outputs.
- Spot-check the judge. Sample disagreements with your own eyes; judges can be gamed by verbose or sycophantic answers if the rubric is soft.
- Save the eval run. Rubric + scores + raw answers are the artifact — not a Slack screenshot.
Rubric patterns that work
- Structural: schema, markdown rules, language constraint
- Grounding: “only use facts present in the context block”
- Safety / policy: explicit refuse conditions
- Product tone: reading level, length, banned phrases
- Tool readiness: whether a supposed function call is well-formed (when you evaluate that layer)
Avoid rubrics that only say “best answer wins.” The judge needs something to measure.
Provider-specific notes
- OpenAI — confirm the model id you intend to ship; playground defaults drift from API ids.
- Anthropic — message and system prompt shapes differ; do not reuse an OpenAI body blindly.
- Gemini — endpoint and auth differ again; treat it as its own provider template.
- OpenRouter / compatible hosts — verify the upstream model actually served; ids can look familiar while routing elsewhere.
Desktop vs CI
Interactive judging on macOS is ideal while the rubric and prompt still move. When both stabilize, graduate cases into a CI framework (Roshi vs Promptfoo) so every PR re-runs them.
Do both in Roshi
Roshi combines direct provider calls, parallel comparison, and an LLM judge with your rubric — local-first on macOS, with no Roshi-operated proxy between you and the providers.
FAQ
Which model should be the judge?
Prefer a capable model you trust for instruction following, ideally not identical to every candidate under test when you can avoid it. Keep the judge fixed across a bake-off so scores stay comparable.
What belongs in the rubric?
Observable criteria: required fields, factual constraints, tone rules, refusal conditions, length limits. Avoid vague “be helpful” language the judge cannot apply consistently.
Is LLM-as-a-judge enough for production gates?
It is a strong exploratory and triage signal. For CI, encode stable cases in an automated framework and treat judge scores as one assertion type among others — see Roshi vs Promptfoo.