Test results

Compare speed, quality and configuration across runs on one desktop. The quality tests differ between rows; open Notes for the conditions.

11 results

Measured results

Select a column heading to sort; select it again to reverse the order. Eight-stream speed is total throughput. Configurations and quality tests differ between runs.

Swipe the table sideways for more columns →

Historical runs from one desktop. Activate a column button to sort. Expand Notes for each run's conditions.
2026-10-01 Two B60s · official FP8 · host-staged evaluation
Setup and notes

Qwen3.8-27B
FP8 (official pre-quantized, block-wise e4m3, 128x128 scales)
2x Intel Arc Pro B60 24GB (TP=2, P2P allreduce, host-staged for this eval)
vLLM (intel/llm-scaler-vllm:0.26.0-b2)
0.26.1.dev0+g568afb3a1
Context: 262144

Candidate, not adopted: Qwen's pre-quantized FP8 release vs. our own online (at-load) FP8 of the same BF16 weights. Roughly tied on speed and quality; switch rule (adopt only if it beats production on both LCB and speed within 5%) was not met.

53 227
10/10

correctness gate

2026-10-01 Two B60s · AutoRound Int4 · host-staged evaluation
Setup and notes

Qwen3.8-27B
AutoRound Int4 (community build, relabelled as GPTQ for the fast kernel path)
2x Intel Arc Pro B60 24GB (TP=2, P2P allreduce, host-staged for this eval)
vLLM (intel/llm-scaler-vllm:0.26.0-b2)
0.26.1.dev0+g568afb3a1
Context: 262144

Best-scoring int4 candidate found in the int4 quant search (Frozenlock/Qwen3.8-27B-int4-AutoRound). Did not match the FP8 reference on quality at greedy decoding, so it was kept as the int4 rollback reference rather than switched to.

73 317
6/9 (vs FP8 7/9) / 10/12 / 10/10

LiveCodeBench-9 (greedy) / exact-answer(12) / gate

2026-10-01 Two B60s · FP8 · vision tower disabled
Setup and notes

Qwen3.8-27B
FP8 (online, at load, official BF16 weights)
2x Intel Arc Pro B60 24GB (TP=2, P2P allreduce)
vLLM (intel/llm-scaler-vllm:0.26.0-b2)
0.26.1.dev0+g568afb3a1
Context: 262144

The checkpoint carries an unused(?) vision tower that vLLM profiles and warms up on every start. --language-model-only cut engine start from ~230s to 131s and freed KV capacity; adopted (production start since is ~142s). Whether anything actually sent this model images could not be confirmed (gateway logs bodies, not metadata, only).

Not measured Not measured
226-239s / 396,601 tokens -> 131s / 452,706 tokens (+14%)

engine start time / KV pool (vision tower on vs. off)

2026-09-30 Two B60s · online FP8 · host-staged fallback
Setup and notes

Qwen3.8-27B
FP8 (online, at load, official BF16 weights)
2x Intel Arc Pro B60 24GB (TP=2, host-staged allreduce, P2P tripped)
vLLM (intel/llm-scaler-vllm:0.26.0-b2)
0.26.1.dev0+g568afb3a1
Context: 262144

Automatic fallback path after a real GPU engine-reset fault tripped the P2P watchdog, which restarts serving on the validated host-staged config. Same checkpoint as the P2P FP8 row below; P2P vs host-staged is the clean comparison here.

37 155 Not recorded
2026-09-30 Two B60s · online FP8 · P2P
Setup and notes

Qwen3.8-27B
FP8 (online, at load, official BF16 weights)
2x Intel Arc Pro B60 24GB (TP=2, P2P allreduce)
vLLM (intel/llm-scaler-vllm:0.26.0-b2)
0.26.1.dev0+g568afb3a1
Context: 262144

Selected for production on 2026-09-30. Chosen over the int4+int4-head checkpoint above: FP8 solved a strict superset of int4's LiveCodeBench problems (14/24 vs 12/24) at about half int4's tokens/turn, at the cost of slower decode and roughly half the KV pool (395K vs 757K tokens).

52.9 251
14/24 / 75 / 9 pass, 0 fail, 1 n/a

LiveCodeBench-24 (medium thinking) / websites (/75) / readiness (10 tasks)

2026-09-29 Two B60s · Int4 weights, head & MTP layer · P2P
Setup and notes

Qwen3.8-27B
GPTQ-Int4 (sym, g128) + int4 lm_head + int4 MTP layer
2x Intel Arc Pro B60 24GB (TP=2, P2P allreduce)
vLLM (intel/llm-scaler-vllm:0.26.0-b2)
0.26.1.dev0+g568afb3a1
Context: 262144

lm_head (2.54 GB FP16) is read k+1 times per decode step (~40% of bytes moved); quantizing it plus the MTP layer to int4 was the lever left once a graph-capturable 2-rank allreduce (vLLM PR #54768, measured +1.7% by its author) wasn't worth building. 55% of the 456 GB/s spec bandwidth roofline / 63% of the achievable 399 GB/s.

86.3 342
156/164 / 240 (n.s. vs 243) / 10/10 / 158

HumanEval / GSM8K(250) / gate / exact-recall(160)

2026-09-29 Two B60s · Int4 · P2P · MTP k=4
Setup and notes

Qwen3.8-27B
GPTQ-Int4 (sym, g128, BF16 MTP head)
2x Intel Arc Pro B60 24GB (TP=2, P2P allreduce)
vLLM (intel/llm-scaler-vllm:0.26.0-b2)
0.26.1.dev0+g568afb3a1
Context: 262144

k=4 vs. production k=3 (70.2 single / 319 at 8 on this same checkpoint). k=3 was kept. k=2 was also tested and rejected (along with full-graph decode and an in-graph drafter) but no speed number was recorded for it in the source.

67 311
k=4

MTP speculative tokens (k)

2026-09-28 One B60 · Int4 · eager
Setup and notes

Qwen3.8-27B
GPTQ-Int4 (sym, g128, BF16 MTP head)
1x Intel Arc Pro B60 24GB
vLLM (intel/llm-scaler-vllm:0.26.0-b2)
0.26.1.dev0+g568afb3a1
Context: 81920

Single-card baseline. Eager mode; measured at 85% of the card's 29.4 tok/s bandwidth roofline.

23.9 160.6
10/10

correctness gate

2026-09-28 Two B60s · Int4 · host-staged · eager
Setup and notes

Qwen3.8-27B
GPTQ-Int4 (sym, g128, BF16 MTP head)
2x Intel Arc Pro B60 24GB (TP=2, host-staged allreduce)
vLLM (intel/llm-scaler-vllm:0.26.0-b2)
0.26.1.dev0+g568afb3a1
Context: 262144

Naive TP=2, eager, no graph capture. Slower than one card: the Alder Lake host bridge (8086:4650) is not on the kernel's p2pdma whitelist, so the allreduce is host-staged and launch overhead on half-size kernels cancels the split.

13.7 102 Not recorded
2026-09-28 Two B60s · Int4 · host-staged · graph + MTP k=3
Setup and notes

Qwen3.8-27B
GPTQ-Int4 (sym, g128, BF16 MTP head)
2x Intel Arc Pro B60 24GB (TP=2, host-staged allreduce)
vLLM (intel/llm-scaler-vllm:0.26.0-b2)
0.26.1.dev0+g568afb3a1
Context: 262144

PIECEWISE CUDA-graph-style capture with vllm::all_reduce as a split point, plus MTP k=3 with the drafter forced eager (in-graph drafter collapses acceptance to 1.01). Validated run 253-validate-20260928-082853; 900s stress 64/0 fail.

42.3 172.5
10/10 / pass

correctness gate / tool call

2026-09-28 Two B60s · Int4 · P2P · graph + MTP k=3
Setup and notes

Qwen3.8-27B
GPTQ-Int4 (sym, g128, BF16 MTP head)
2x Intel Arc Pro B60 24GB (TP=2, P2P allreduce)
vLLM (intel/llm-scaler-vllm:0.26.0-b2)
0.26.1.dev0+g568afb3a1
Context: 262144

Same checkpoint and graph config as the row above, with GPU-to-GPU P2P enabled via a kernel whitelist override for the Alder Lake host bridge (out-of-tree kretprobe module). Soak run under a concurrent ZFS scrub on the SATA mirror behind the PCH.

56 301.8
10/10 / pass / 312 ok, 0 fail (900s)

correctness gate / tool call / stress

Context is the configured --max-model-len. Speeds are greedy decode tokens/second, 1 stream and 8 concurrent. Public raw logs for these older rows are not yet available. Reviewed notebook results are in the JSON feed.