vLLM KV cache: check context length and concurrent request capacity
Read the startup token budget, leave room for the answer, and distinguish a model that loads from a workload that runs reliably.
A local model can finish loading and still struggle when you add a long document or a second user. Before choosing a smaller model or another GPU, separate three questions: do the weights fit, does the intended request fit, and can several requests finish at an acceptable speed?
This guide explains how to read vLLM's capacity information and build a useful troubleshooting record. It uses vLLM 0.32.0 documentation. The arithmetic below is illustrative; we have not measured a new capacity limit or speedup on our Intel GPU machine.
Read the two startup figures together
In its scaling guide, vLLM describes a total GPU KV-cache token count and a concurrency estimate at a stated sequence length. KV cache holds attention state used during inference. The total is shared capacity, not a context window promised to each simultaneous user.
Keep the sequence length beside the estimate. A figure calculated for short requests does not describe the same workload as several long conversations.
Include room for the answer
The maximum model length covers prompt and output. Count the complete formatted input, including instructions, conversation history and retrieved text, then allow for the answer. A document that nearly fills the configured limit leaves little room to respond.
For a troubleshooting record, write these as separate quantities. That makes it easier to notice when an application sends more history than you intended.
| Part | Tokens |
|---|---|
| Formatted prompt, history and document | 6,144 |
| Planned answer allowance | 2,048 |
| Total planned sequence | 8,192 |
Calculate a rough cache budget, then test the workload
Suppose an imaginary server reports capacity for 32,768 cached tokens. Four independent sequences of 8,192 tokens would consume that simple token budget: 32,768 divided by 8,192 equals four. This example assumes no shared-prefix saving and treats every sequence as reaching the full planned length.
It does not establish that four users will receive fast answers. Nor is it a way to predict a real model's cache size from the memory printed on a GPU box. Read the actual engine's reported capacity and configuration; different models, cache layouts and supported settings need their own record.
Use the calculation to frame a question: what happens when this particular request pattern overlaps? A useful test keeps input lengths, answer allowance and model configuration fixed, increases concurrency in small steps, and retains the slow and failed requests. Record first-token delay, completion time and answer correctness alongside the number of completed requests.
Match the symptom to the evidence
| Symptom | Record first | Question to investigate |
|---|---|---|
| The server fails during startup | Exact error, stack version, model artifact and configured length | Did weight loading, cache allocation or another initialization step fail? |
| A long request is rejected | Actual formatted input tokens, answer allowance and request error | Does the complete requested sequence exceed the configured limit? |
| One request works; overlapping requests slow down | Arrival times, first-token/completion delays and cache-pressure counters | Are queueing or memory pressure consistent with the change? |
| The logs report preemption | Counter changes over the test and the matching request ledger | Does a smaller admitted workload improve completion behavior? |
A symptom identifies an investigation, not a diagnosis. For example, a slow first token does not by itself prove that the cache is full.
Interpret preemption without guessing its cost
vLLM's tuning guide explains that insufficient cache space can lead to request preemption and later recomputation, affecting latency. It lists lower batching/concurrency limits and more available cache among the options to investigate. Those controls have different effects; do not change several at once and credit the result to one of them.
A preemption counter helps identify cache pressure. It cannot, alone, tell you how many milliseconds came from recomputation rather than queueing. Keep counter changes aligned with the test interval and client timings.
Save a record another person can reproduce
Keep the model revision and precision, serving image or build, GPU arrangement, effective context limit, both startup capacity figures, input/output counts, concurrency and errors together. Record whether prompts reuse an identical beginning; our prefix-caching guide explains why that is a separate comparison condition.
Change one supported setting at a time in a controlled test, then repeat the same requests. Protect other workloads on a shared machine. The useful outcome is a configuration that finishes the intended work reliably within your response-time budget—not the largest context number that can be entered.
For the next comparison, use our inference measurement checklist. If you are setting up a dual-Intel-GPU server, start with the setup reference and match its tested conditions to your own.