Tokens per second don't tell the whole story
Chat, coding agents and batch jobs need different measurements. Here's a checklist for comparing local inference setups.
A model can stream text quickly and still leave you waiting for an answer. Tokens per second measures one part of the experience. It doesn't tell you how long the model took to start, or whether the answer finished the job.
Before changing a local inference setup, choose the problem you're trying to improve. A chat session, several coding agents and an overnight batch need different measurements.
Measure the part you actually wait for
| Workload | What to record |
|---|---|
| Interactive chat | Time to first token, gaps between tokens, total response time |
| Coding agents | Time to finish the task, pass/fail result, slow requests |
| Batch processing | Successful jobs per hour, failures, total elapsed time |
The vLLM metrics documentation distinguishes the wait for the first token from the time between output tokens. Keep both when testing an interactive workload. A fast stream after a long pause still feels like a long pause.
Keep the job the same
Use a fixed set of prompts and a written definition of a successful answer. Keep the model revision, input lengths, output limit, sampling settings and number of simultaneous requests consistent. Record whether the cache is warm.
Then change one setting. Separate warm-up runs from measured runs, repeat the comparison, and keep the failures. Extra tokens aren't a useful improvement if the answer fails the task.
Keep the evidence, too
Save the command, software versions, prompt identifiers, raw timings and task verdicts. Include host memory use and startup behavior when changing graph capture or speculative decoding. That makes it easier to revisit the choice when the workload changes.
This is a proposed measurement checklist, not a new benchmark. For a measured example, see the graph-capture comparison: the gain in decode speed was larger than the gain on complete agent tasks.