Roshi vs Postman for LLM APIs

Postman earned its place as the default API client. LLM HTTP APIs look like “just another POST” until you live in streaming responses, multi-turn message lists, and vendor-specific auth for a week.

What Postman is great at

• Arbitrary HTTP: any method, auth helper, environment variable, and pre-request script.
• Team collections and monitors for conventional REST backends.
• Broad protocol coverage beyond chat completions.

If you are debugging your own /v1/invoices API, stay in Postman.

Where LLM traffic gets awkward in a generic client

Streaming. Server-sent chunks need a UI that accumulates tokens and still shows the raw event stream when something breaks.

Message editing. System / user / assistant / tool turns are not a single JSON textarea you want to reformat by hand every attempt.

Provider drift. OpenAI, Anthropic, and Gemini disagree on URLs, headers, and body shapes. A template per provider beats one overstretched collection.

Comparison. “Run this across three models” is an eval feature, not an HTTP verb. Postman can script it; you will build the product yourself.

Local secrets for AI work. Prompt text often includes customer-like data. A local-only client reduces how many sync targets that data touches.

Side-by-side

Job	Prefer
General REST / GraphQL	Postman (or similar)
One-off LLM POST to learn an API	Either
Daily streaming chat debugging	Roshi
Multi-provider model bake-off	Roshi
LLM-as-a-judge on a live run	Roshi
Company-wide non-LLM collection sync	Postman
CI prompt regression suite	Promptfoo-class tools (comparison)

A practical split

Keep Postman for infrastructure and product HTTP. Keep a local LLM API client for the model layer. When the question shifts from “does this request 200?” to “which model should we ship?”, use multi-model comparison instead of duplicating the collection three times.

Try Roshi

Roshi is a local-first macOS workbench for LLM API testing and evals: direct-to-provider requests, streaming inspection, parallel model runs, and optional judging — without turning your prompts into another cloud workspace.