vLLM prefix caching: shared prompt reuse and first-token latency
Reuse a document or conversation prefix to save prompt processing. Here is how to tell whether that matches your workload—and how to compare it fairly.
Asking several questions about the same long document can make your local model process much of the same input repeatedly. Prefix caching can reuse that earlier work. The useful question is whether your requests share a beginning, and whether processing that beginning is a meaningful part of the wait.
This is a practical explainer, not a new benchmark. The behavior described here is documented for vLLM 0.32.0. We have not measured a prefix-caching gain on the two-B60 machine for this article.
What gets reused
vLLM calls the initial processing of an input prefill. Its automatic prefix cache retains reusable attention state, often called KV cache, from earlier requests. A matching beginning can avoid repeating that portion of prefill. Generating the new answer is a separate phase: a cache hit does not directly accelerate each newly generated token. See the versioned feature guide.
A prefix means the beginning of the token sequence, not just similar meaning. vLLM's cache design uses block tokens and the preceding prefix to identify reusable blocks. It caches full blocks. Two prompts about the same subject are not, by themselves, evidence of a reusable prefix.
A useful prompt pattern
Consider a small documentation assistant. Each request begins with the same instructions and manual; a different question follows. That is a candidate workload, not a promise of a particular speedup.
Request A: [fixed instructions] [same manual] [question A]
Request B: [fixed instructions] [same manual] [question B]Compare that with an application that rewrites the instructions or inserts a changing value at the beginning of every request. Inspect what the application actually sends, including its chat template, before assuming the visible document is the whole prefix. Keep sensitive prompt content out of public logs.
vLLM names repeated document questions and continuing conversations as example workloads. Unrelated inputs offer less reuse, while a long generated answer can still dominate the total time. Prefix caching is therefore a workload choice, rather than a general cure for slow inference.
Write down the comparison before running it
The following is a suggested test plan. None of these rows represents a result collected by LocalXPU.
| Case | Purpose | Keep in the record |
|---|---|---|
| First use of a fixed prefix | Establish the initial request | How cache state was established; input and output lengths |
| Repeat the prefix with another question | Look for reuse across requests | The actual common prefix and available cache-hit evidence |
| Use a different prefix | Check whether improvement depends on reuse | The changed input; report this as a separate workload |
For an on/off comparison, keep the checkpoint, precision, serving build, prompt set, output limit, sampling and concurrency fixed. Establish and document the intended cache state before each case; do not label a warmed repeat request as a cold first request. Repeat the cases, retain failures and record their order. Separate compilation warm-up from cache reuse so the two effects do not get credited to one feature.
Measure the wait, not just the token rate
Record time to first token, total response time and whether the answer passes your task's check. Also keep prompt and output lengths. A faster start with an incorrect answer is not a useful improvement.
For vLLM 0.32.0, the metrics documentation lists vllm:time_to_first_token_seconds, vllm:e2e_request_latency_seconds, and prefix-cache query and hit counters. Use counter changes over the relevant test interval as supporting evidence. They describe cache activity; they do not, alone, establish a per-request latency saving. Other traffic can contaminate a shared server's counters, so isolate a comparison through your normal scheduler.
If first-token latency improves but end-to-end time barely changes, report both. If the application becomes quicker without a documented cache hit, investigate before attributing the change. Our measurement checklist covers the broader record to keep.
Check your version before changing settings
Use the documentation for the engine you actually run. This article is not a recommendation to upgrade an older Intel container to 0.32.0, and its general explanation does not establish support for every model or execution mode. Check the model-specific requirements and retain a working configuration before experimenting.
For a shared service, also review the cache-isolation guidance before deciding which requests may share state. The design documents request salts for separating reuse groups; that mechanism is not a replacement for access control.