Measure the serving system before enabling CUDA graphs.
Qwen3.8-27B studies · Configuration study
Beyond
tokens per second.
This study measures inference optimization as a joint performance-and-behavior problem: throughput, latency, memory, correctness and answer stability under the same deployment conditions.
THE QUESTION
How do choices—from quantization and CUDA graphs to speculative decoding, concurrency and prompt design—affect throughput, latency, correctness and answer stability?
WHY RV-380
RV-380 is a generated set of 380 mixed questions designed to approximate a varied application workload and reduce benchmark-specific tuning effects. It compares inference configurations; it does not rank general intelligence.
WHAT WE CHECKED
We compared four weight-quantized checkpoint builds—AWQ, GPTQ, AutoRound and Red Hat AI’s llm-compressor build—plus FP16 and FP8 KV caches, MTP and n-gram speculative decoding, concurrency, prompt design and reasoning prompts. We checked aggregate scores and individual answers.
Qwen3.8-27B was tested across 71 controlled inference configurations on the RTX 4090 GPU family. The model and RV-380 requests stayed fixed while the inference stack changed. Unless a finding says otherwise, every result uses standard non-speculative autoregressive decoding: one token generated at a time, with no speculative decoding. The two exceptions — MTP and n-gram speculative decoding — are clearly marked.
EXPERIMENT MAPMEASURED
FIXEDQwen3.8-27BNvidia RTX 4090 GPU family · 24 GiB
71 configurations
CHANGEDInference stack
WEIGHT-QUANTIZED BUILDS
AWQGPTQAutoRoundRed Hat AI
EXECUTION
EagerCUDA graphs
KV CACHE
FP16FP8
SPECULATIVE DECODING
None (control)MTPn-gram
REQUEST + LOAD
ConcurrencyPrompt designReasoning prompts
PERFORMANCESpeed + waittok/s · TTFT · memory
BEHAVIORQuality + drift380 scored questions
same modelsame questionspaired comparison
71configurations served
380fixed, scored tasks
3.08×CUDA-graph speedup
−18.9largest quality shift, points
3–4natural correctness flips / 380
THE SHORT VERSION. KEY FINDINGS FROM THE ANALYSIS DETAILED BELOW.
Five things we would tell you in an elevator.
- Turn CUDA graphs on. Three times faster for a single user — 16.1 to 49.6 tokens per second — and with no measurable loss in accuracy.
- AWQ is the robustness default. Of the four builds tested, it is the only one never significantly beaten on any scoring basis or answer regime — which is not the same as scoring highest. On one strict long-answer view it ranks third by mean and still is not beaten. Pick it because nothing separates from it, not because it wins.
- Leave n-gram speculative decoding off. It got 18.2 points fewer questions right and ran 2.6× slower. There is no case for it in this setup.
- Treat MTP as model & workload-dependent, not a free speedup. On this model, in our tested stack, enabling MTP produced material answer changes that varied sharply with the response regime — from −11.9 points on short answers to −1.2 on long-form reasoning. The effect was substantially larger than our measured repeat-run variation. We would validate MTP against the target workload before deploying it; a dedicated follow-up examines the mechanism in detail. The technical note →
- Watch the wait, not just the throughput. Total throughput keeps climbing as users arrive, but the wait before anything appears goes from 0.32 s to 28.4 s. Past about eight users per GPU, most of a request's life is spent queuing.
Everything below is the evidence for those five lines. All of it was measured with the model's own reasoning mode switched off; turning it on increases accuracy, but also increases token count and is worth a further 4.4 accuracy points. It is comprehensively covered in our reasoning study →
Plain-English glossary Every term this page uses, defined once. Click to open.
Speed and waiting
- Token
- A word or piece of a word. Models read and write in these.
- Tokens per second (tok/s)
- How fast the model writes once it has started.
- Time to first token (TTFT)
- How long you stare at nothing before the answer starts.
- Prefill
- The phase where the model reads your prompt.
- Decode
- The phase where the model writes the answer.
- Queue
- Waiting your turn while the server serves somebody else.
- Concurrency
- How many people are asking at the same moment.
The inference stack
- Quantization
- Storing the model's weights at lower precision so it fits and runs faster.
- Checkpoint build
- One downloadable copy of the model, quantized a particular way.
- KV cache
- The model's working memory for a request. It lives in GPU memory and limits how many people fit.
- CUDA graphs
- Replaying the GPU's work as one recorded sequence instead of issuing it step by step.
- Autoregressive decoding
- The ordinary way a model writes: one token generated at a time, each chosen after seeing the last. The baseline for everything on this page.
- Speculative decoding
- Guessing several tokens ahead, then keeping only the guesses that check out. Faster when the guesses are right.
- MTP
- The model's own built-in guesser for speculative decoding. Every MTP result on this page describes Qwen3.8-27B on this stack, not MTP in general.
- Speculative depth
- The number of tokens the predictor guesses ahead in a single step, on top of the autoregressive token the model produces anyway. Depth 3 means one autoregressive token plus up to three guesses, kept only if verification accepts them — so depth is the ceiling per step, not the yield. Greater depth means a larger speed win when the guesses are right, and more work discarded when they are not.
- n-gram
- A guesser that copies repeated phrases out of the prompt.
- Preemption
- The server evicting a half-finished request to make room, then redoing it later.
How we measure
- RV-380
- Our fixed set of 380 questions, marked right or wrong by code.
- Greedy decoding
- The model always takes the most likely next token, so sampling randomness is removed. It does not make GPU inference bit-for-bit deterministic — identical re-runs still move 3–4 answers per 380 — but it removes the dice, which is what lets a difference between runs be attributed to the setting we changed.
- Paired comparison
- Comparing two setups question by question rather than score against score. Only the questions they disagree on carry information.
- p-value
- If the two setups were genuinely equally good, the probability of seeing a split at least this lopsided. Small means the result is hard to explain by chance. Above 0.05 we report that this experiment did not separate them — which is not the same as showing they are identical.
- Noise floor
- How much results wobble between two identical runs: 3–4 questions in 380. Anything smaller than that is not a finding.
The all-important tokens per second
There isn’t one
tokens-per-second
number for a model.
The same model on the same GPU reports wildly different tokens per second (tok/s) depending on which phase you time, whose rate you mean and how many people are asking at once. Before any result on this page means anything, it helps to know which of these is being quoted. The figures below are round illustrations, not our measurements.
What is being measured
Prefill throughput — input tokens per second (input tok/s)
How quickly the model reads the input prompt, before generation begins.
Prefill is much faster than generation, because the whole prompt is processed in parallel rather than one token at a time. Input rates in the thousands of tokens per second are normal while output rates sit in the tens. Reading the question is rarely the expensive part.
Time to first token — TTFT, measured in seconds
Time from submitting the request until the first output token appears.
On an idle server this is essentially prefill. On a busy one it is mostly queueing — time the request spends waiting its turn before any work starts. TTFT fuses the two, so on its own it cannot tell you whether the model was slow or the server was simply full.
Decode throughput — output tokens per second, per request
How quickly a single request produces tokens after the first one. It is the inverse of TPOT (time per output token), the average gap between successive output tokens: decode tok/s = 1 / TPOT.
That relation holds at any concurrency. What load changes is TPOT itself — as more requests share the GPU, the gap between one user’s tokens stretches, so their decode rate falls.
End-to-end throughput — output tokens per second, per request
Output tokens divided by the whole request time — queueing, prefill, decode and runtime overhead together.
Always lower than decode throughput, because it is diluted by the phases that produce no tokens. How much lower depends entirely on the shape of the request.
Performance under load — one user vs many concurrent users
Peak single-user performance — tokens per second at concurrency 1
The fastest experience one active request can get, with the whole GPU to itself.
This is a latency measure. It describes what one person waiting at a screen sees, and it is the best case — every additional concurrent user makes it worse.
Maximum server throughput — total tokens per second, all requests
How many tokens the server produces in total when the GPU is shared across many concurrent requests.
This is a capacity measure. Raising concurrency lifts the server’s combined tokens per second while every individual user gets fewer and waits longer. The two move in opposite directions, which is why one number cannot stand in for the other.
Per-user and server-wide rates count different things
Suppose one user alone gets 50 tok/s, and the same server running six concurrent requests produces 200 tok/s in total. The second number is not four times faster — it is six people sharing, each receiving about 33 tok/s. Quoting a server-wide figure as though it described one user’s experience is the most common way a tokens-per-second claim misleads.
What the workload changes — traffic pattern matters
Prompt and answer length decide which number matters
Short prompt, long answer — for example, "Tell me a story of a 1000 words". Almost all the time goes on decode, so end-to-end tokens per second sits close to the decode rate, and decode speed is what your users feel.
Long input, short answer — summarising or classifying a large document. Now prefill dominates and the answer is over in a moment, so end-to-end tokens per second collapses well below the decode rate even though the model generates at exactly the same speed. Here TTFT and prefill throughput are the numbers that matter, and a decode-rate headline tells you almost nothing.
BOTTOM LINE
A tokens-per-second figure means nothing on its own. You need to know which phase it times, whether it is one user or the whole server, at what concurrency, and on what input and output lengths. You also need to be very clear on whether speculative decoding is being used.
The serving baseline
First, establish
the vLLM baseline.
The 3.08× speed-up was measured after establishing a working vLLM baseline. It is an improvement within vLLM. Using vLLM’s published 14–24× throughput comparison with a basic Hugging Face Transformers loop as external context, the combined estimate is roughly 60× over the simple implementation. The calculation and its limits are shown below.
Change the execution mode while keeping the model, GPU and single-user workload fixed.
Both figures above are end-to-end, so the 3.08× compares like with like. Strip out prefill and the CUDA-graphs arm decodes at 53.2 tok/s — the rate tokens actually arrive at once generation starts. Where a request’s time goes →
Why this experiment
Optimization is a
multi-axis decision.
Faster generation is useful only when the model still behaves as the product requires. We measured speed and answer behavior together for every optimization.
THE FULL 4K VIEWfour key lenses · one deployment decision
01 · SPEEDHow fast?tok/s · throughput
02 · WAITHow responsive?TTFT · queue time
03 · CAPACITYHow much fits?memory · concurrent users
04 · BEHAVIORWhat changed?correctness · answer drift
A TYPICAL GPU REPORT
- 01How fast?
Peak tokens per second.
- 02Does it fit?
Model and cache memory.
Necessary measurements. Not a deployment decision.
THE RADIANVECTOR STUDY
- 01How fast—and for whom?
Generation rate, aggregate throughput, time to first token and queueing under load.
- 02Does behavior survive?
Correctness, exact output changes and question-level flips under each configuration.
- 03Is the conclusion stable?
Repeated boots, paired tests, multiple answer regimes and a measured noise floor.
MODELQwen3.8-27B W4A16
RUNTIMEvLLM 0.28 · WSL2
HARDWARERTX 4090 GPU family · 24 GiB
QUALITY INSTRUMENTRV-380 · 17 scored task types
DECODINGGreedy* · native thinking off
* Greedy means temperature 0: at each step the model takes the single highest-probability next token instead of sampling among the plausible ones. That removes the sampling randomness — though not every source of variation, since identical re-runs still move 3–4 answers in 380 — which is what makes a difference between two runs attributable to the configuration rather than to chance. How we read these numbers →
WHAT RV-380 COVERS
Five capability bands, checked by code.
The suite is a regression instrument for inference changes—not a claim to measure all of intelligence.
The behavior check
RV-380,
question by question.
Every configuration is evaluated on the same deterministic questions, with each response checked programmatically. For deployment work, the same method can use an evaluation set built around the application’s domain, use cases and traffic.
380questions scored
17scored task types
5capability bands
fixedsame suite for every run
ALL 17 RV-380 TASK TYPES
See exactly what was tested.
WHAT FUTURE VERSIONS SHOULD ADD
A focused regression suite—not a complete measure of model capability.
RV-380 includes 25 simple general-knowledge questions about capitals, chemical symbols and basic facts. That is useful for checking one-word instruction following, but it is not broad knowledge coverage.
Future versions should expand general knowledge, safety, multilingual and multimodal evaluation. Code execution was deliberately left for a separate RadianVector benchmark and report focused on coding models, executable tests and development workflows.
01Broader knowledgeMore subjects, difficulty levels and current factual domains
02Code executionA dedicated coding benchmark and report is planned
03SafetyRefusal quality, boundary handling and robustness
04MultilingualInstruction following and reasoning across languages
05MultimodalImage and mixed-input understanding
Experimental design
How to read
these numbers.
01 · GREEDY DECODING
The model never rolls a die.
Greedy means temperature 0. At each step the model takes the single highest-probability next token, instead of sampling among the several plausible ones. That removes the sampling randomness. It does not make the serving stack bit-for-bit deterministic — we measure that floor below, and it is 3–4 answers in 380 — but it takes the dice out of the comparison.
That is not how most products run — a little sampling usually makes output feel less mechanical. It is how a measurement has to run. With sampling on, two runs of one configuration already differ, and there is no way to tell a real effect from the dice. Greedy makes the configuration the only moving part, so a difference between two runs is attributable to the change we made.
Under greedy decoding, speculative decoding is close to output-preserving by construction: the verifier accepts a drafted token only when it matches what the model would have picked. vLLM qualifies that guarantee around hardware numerics and does not guarantee stable output across runs because batch size can change the arithmetic. Small divergences are therefore expected.
What we measured on this model is far bigger than that. In the short-answer regime, MTP at depth 3 lost 144 questions and won only 8, while re-running the same configuration twice moves three or four answers. A speed-only benchmark would have recorded this configuration as a win. Technical note →
02 · PAIRED COMPARISON
Question by question, not score against score.
Each comparison walks all 380 questions and sorts them into four groups: both right, both wrong, only A right and only B right.
The questions where A and B agreed are then set aside. One they both answer correctly tells you nothing about which is better, and neither does one they both fail. Only the disagreements carry information; the test asks whether one side won them more often than chance would explain.
That is why a line on this page can read paired 3 vs 4: the two arms disagreed on seven questions in total, one winning three and the other four. A coin flip. No difference is claimed, however far apart the raw percentages look.
RV-380 contains questions at different difficulty levels. Pairing controls for that mix because both arms face the identical question. It can reveal small consistent effects and show when a large-looking score difference rests on only a few items.
Where a request’s time actually goes
Both settings above are about correctness. This one is about what the server feels like when more than one person is using it — and it is the most counter-intuitive thing we measured. Each bar is one request’s whole life, split into the four phases the server actually reports: time queued before anything starts, time spent reading your prompt, time spent writing the answer, and the scheduling remainder. The only thing that changes down the rows is how many people are asking at once.
ONE REQUEST’S LIFE UNDER LOAD · FOUR MEASURED PHASES · THINKING OFF
queued — not being served at all
prefill — reading the prompt
decode — writing the answer
other — scheduling remainder
hover any segment for its seconds and share
Load does not make the model slower. It makes you wait longer to start. Decode — the only phase that produces text — takes 4.78 s at one user and 7.72 s at thirty-two, a factor of 1.6. Queue goes from nothing at all to 22.40 s. At eight users the request is still only 3% queued; by thirty-two it is 72%, and the machine is working the whole time on somebody else’s request.
Prefill behaves differently again, and it is the one phase that gets better under load: 0.35 s alone, rising to 1.24 s at six users, then falling back to 0.77 s at thirty-two as the scheduler batches prompts together. It never dominates. Reading the question was never the expensive part.
This split also reconciles the two generation-rate figures in this project. At one user, decode is 93.1% of the whole request — 4.783 s of 5.137 s, with prefill 0.352 s and no queue at all. The 49.6 tok/s quoted above for CUDA graphs is an end-to-end rate, so it is diluted by that prefill. Divide it out — 49.55 ÷ 0.931 — and you get 53.2 tok/s of pure decode, which is exactly where the reasoning study’s decode-only range begins. The two numbers match — they are the same speed, measured two different ways.
Throughput on its own does not tell you what the wait was. The same server can post a higher tokens-per-second figure while every individual user waits longer for their answer.
Config M1_mamba_bf16, CUDA graphs on, 64 scheduler slots, 27,306-token KV pool. Synthetic 1,024-token prompts with a 256-token cap, three rounds per load level, so the phase split is not confounded by questions of different lengths. The phases are the server’s own queue_s, prefill_s and decode_s; “other” is the residual against end-to-end.
Two consequences worth stating plainly. Pairing only works if both arms saw the same questions in the same rounds — which is why results from different suite hashes are never pooled, and why a comparison drawn across two campaigns is reported as such rather than merged. And because throughput climbs with load until the server saturates, with different configurations peaking at different points, any two configurations compared later on this page are read at the same number of simultaneous users. Taking each at its own best load would reward whichever one peaks earliest.
Executive results
Eight experiments.
Four decisions.
Filter by the kind of decision each experiment informs. Every card distinguishes the observed result from its operational interpretation.
01USE
CUDA graphs
3.08×single-user generation
16.1 tok/s 49.6 tok/s
Accuracy was unchanged on the stated 380-item basis when re-tested under the harder reasoning regime: 95.18% versus 95.26%, paired 3 vs 4, p=1.0.
02PREFER AWQ
Four 4-bit checkpoint builds
2.4points between builds*
AWQ is the only build never significantly beaten across every tested scoring basis and both answer regimes. The order beneath it is not robust.
*These are four downloadable builds, not four independent algorithms. The Red Hat AI recipe combines AWQModifier and GPTQModifier. On the primary 380-item long-answer basis, the range is 2.4 points.03VALIDATE ON WORKLOAD
MTP-head speculative decoding
−11.9 → −1.2accuracy points
The model’s built-in MTP head proposed future tokens; no separate draft model was used. In our tested MTP configuration for this model, it was not behavior-preserving. The effect depended strongly on the answer regime: depth 3 produced the largest shift on short answers and a much smaller one under full reasoning. The change was well above our measured repeat-run variation, so we would gate MTP on workload-level evaluation rather than assume it is a free optimization. Technical note →
Qwen3.8-27B W4A16 · vLLM 0.28 · WSL2 · on the RTX 4090 GPU family. The published follow-up traces the loss to a defect in vLLM's V1 model runner.04AVOID
n-gram speculative decoding
−18.2 ptsand 2.6× slower
The n-gram prompt-lookup proposer used no draft model. It was already unsafe at depth 1 and degraded further with depth under long-form answering.
05NO QUALITY EFFECT FOUND
Cache preemption
3–5×more evictions forced
At matched concurrency, the starved cache produced correctness changes indistinguishable from repeat-run noise: p=1.0 at c=8 and p=0.45 at c=16.
06GATE REQUIRED
Production MTP configuration
+7.4%throughput at 6 users
On this model, MK17 preserved full-reasoning quality (90.09% vs 90.44%, p=0.60) but still lost 6.5 points on answer-only tasks. The throughput gain appeared at six concurrent users; we did not test that value as a safety limit. The workload-level gate in finding 03 applies here too.
07WATCH THE WAIT
Concurrency
0.32s → 28.37smean time to first token
Total throughput rises under load, but responsiveness collapses. A peak-throughput number without TTFT describes the wrong product.
08MEASURE THE FLOOR
Repeatability and audit
3–4 / 380natural correctness flips
Identical runs do not produce identical text. We measured that floor, paired each question, generated figures from raw results and attached every conclusion to its source campaign.
Behind the conclusions
From run to
conclusion.
The polished result is one page. Behind it is a reproducible experiment system: hypotheses, controlled runs, raw telemetry, stored responses, paired quality checks, failed configurations and repeated measurements.
WHAT WE VARIED
More than a speed ladder.
Each technique was tested as a change to a serving system, with performance and behavioral consequences measured together.
THE EVIDENCE ARCHIVE
Results kept at run-level resolution.
The archive retains machine-readable metrics, individual model answers, logs and analysis—not just the winning number.
FROM RUN TO RESULT
How each figure is built.
- 01
Freeze the model, hardware and 380-item scored instrument.
- 02
Repeat runs and compare every question, not only the mean.
- 03
Separate measured relationships from mechanism hypotheses.
- 04
Generate figures from raw results and audit source citations.
- 05
Revise the conclusion, and the page, when a stronger test changes the answer.
WHAT COUNTS AS EVIDENCE
The number of files is not the evidence. The evidence is the traceable chain from a hypothesis, through a controlled run and stored outputs, to a conclusion another engineer can inspect.
Detailed evidence
The numbers behind
the decisions.
01 · QUANTIZATION
A ranking that moves when the scoring basis moves.
The headline table shows four RV-380 views: two answer regimes, each on the full suite and a stricter 360-item subset. “Leader separable” uses a paired McNemar test between first and second by mean. “Never significantly beaten” asks of every build, not just the leader, whether any other build significantly beats it on that basis — so the reader can see whether surviving is a distinction or merely common. AWQ is the only build that survives all four; AutoRound survives three.
What survives all four views
AWQ is the only build never significantly beaten on any of the four RV-380 views; AutoRound survives three and Red Hat AI one. GPTQ is significantly beaten on every view. Nothing below AWQ has a stable ordering. On the strict long-answer subset AWQ ranks third by mean and is still not beaten — “never beaten” means no other build separates from it, not that it scores highest.
| Asked | Counted | Ranking by mean | Leader separable from 2nd? | Never significantly beaten |
|---|
INTERACTIVE CHECKPOINT EXPLORER
Change what counts.
02 · SPECULATIVE DECODING
Answer length changes the MTP trade-off.
Same model, questions and matched control. Only the requested answer style changes. Run time is a proxy for how much text the regime elicited.
Measured vs inferred
The relationship is measured. Longer answers “absorbing” an early divergence is a working hypothesis; this experiment did not isolate that mechanism. The published follow-up traced the loss itself to a defect in vLLM's V1 model runner; why longer answers show less of it was not measured.
| Answer regime | Seconds / pass | Control | MTP depth 3 | Cost | Answers changed | Paired p |
|---|
03 · PROPOSER COMPARISON
MTP-head and prompt-lookup speculation do not fail together.
Both are forms of speculative decoding, and neither uses a separate draft model. At full reasoning, MTP depth 1 ties control while n-gram depth 1 loses 13.1 points. The proposer may drive the gap, but exact verification would prevent either proposer from changing the answer, so the verification path is also part of the mechanism.
Mechanism status
Two explanations fit these data: the proposer itself drives the gap, or it sets how often the rejection path runs, since a weaker proposer is rejected more often. The published follow-up traced both losses to defects in how vLLM restores the model's recurrent state after a speculative step, one affecting MTP and one affecting n-gram.
03b · MTP TECHNICAL NOTE
MTP surfaced a deployment caveat worth investigating separately.
In our tested MTP configuration for this model, MTP was not behavior-preserving: the measured difference ranged from −11.9 accuracy points on short answers to −1.2 points under full reasoning, well above our repeat-run variation.
The operational conclusion
Validate MTP against the actual model and workload rather than treating it as a free speedup. These results characterize Qwen3.8-27B on this stack, not MTP in general.
Why this is notable
Speculative decoding is intended to preserve the target model’s output, subject to numerical and runtime effects. vLLM qualifies its losslessness guarantees around hardware numerics and does not promise bit-for-bit identical output across runs. The divergence measured here was substantially larger than the 3–4 answer repeat-run variation.
The mechanism investigation is published separately: the vLLM MTP bug investigation traces this loss to a defect in vLLM's V1 model runner and tests a fix.
04 · CACHE PRESSURE
We made preemption happen—and still found no quality effect.
The positive control matters. A starved cache demonstrably caused far more preemptions, while concurrency was held constant. That separates cache eviction from the batch-shape changes caused by comparing different concurrency levels.
05 · CONCURRENCY
Throughput and responsiveness pull in opposite directions.
Select or focus a point to inspect the measured mean TTFT and the range across four configurations.
The reasoning study
Thinking on,
thinking off.
Everything above was measured with the model’s reasoning mode off. The reasoning study holds the model, GPU and evaluation set fixed while switching reasoning on.
WHY WE PINNED IT OFF FIRST
- 01One variable at a time
This report changes the serving stack — quantization, cache format, speculation, concurrency. Letting the model also decide how much to think would have put an uncontrolled variable inside every comparison.
- 02A stable floor to measure from
With thinking off the model answers directly and output length is governed by the question, not by an internal budget. That makes the paired comparisons above clean.
- 03It matches how the study began
Every request sent enable_thinking: false. Holding that fixed keeps the whole first body of work internally comparable.
The cost of that choice: none of the numbers above tell you what the model can do when it is allowed to reason.
WHAT THE SECOND STUDY FOUND
- 01Turning it on is worth ~4.4 points
RV-380 accuracy rises from 94.4% to 98.8% averaged across four checkpoint builds — roughly seventeen questions, four to five times the repeat-run noise floor.
- 02The gain is bought at the lowest setting
Almost all of it arrives at low. Every step above that buys 0.4 points or less while generating substantially more text, and the template’s own default is the most expensive rung of all.
- 03It does not rescue speculative decoding
On this model, MTP still costs accuracy with thinking on. The damage tracks how much the model writes, not why it writes it.
Same RV-380 questions, same GPU, same greedy decoding. The only thing that changes is whether the chat template may reason before answering.
STUDY WITH REASONING TURNED ON
The reasoning study covers four checkpoint builds across four reasoning-effort settings: off, low, medium and xhigh. It reports paired significance tests, generated-token cost and throughput.
Thinking on, thinking offDeployment decisions
What we would
actually run.
These recommendations are deliberately narrow: the exact model, runtime, hardware and workload are part of every conclusion.
TURN ON
CUDA graphs
The largest clean speed win: 3.08× single-user generation with no measurable aggregate accuracy cost.
DEFAULT BUILD
AWQ
The only checkpoint never significantly beaten across all tested regimes and scoring bases.
CONDITIONAL
MTP depth 1, long-form only
On this model, at full reasoning, this configuration cost no measurable quality (90.09% vs 90.44%, p=0.60) for +7.4% throughput. On short factual answers the same configuration moved answers materially, so validate against the target model and workload before adopting it and leave it off for short-form traffic. The throughput crossover sat at six concurrent sequences — an observation, not a tested ceiling. Technical note →
LEAVE OFF
n-gram speculative decoding
Worse quality and slower execution in the tested long-answer regime, including at the shallowest depth.
Experimental method
Controls and
comparisons.
A null result needs a positive control, a small difference needs a paired comparison, and every published figure needs a route back to its source run.
Freeze the instrument
Fixed seed, deterministic task generation and a suite SHA. Runs with different hashes are not pooled.
Compare item by item
Every arm answers identical questions. McNemar tests the discordant questions rather than comparing rounded means.
Measure natural variation
Repeated identical runs establish the floor: 3–4 correctness flips and roughly 5% text variation per 380 items. Those repeats all ran at the same batch size, which is the right test for most comparisons on this page but not for one where the batch size itself changes. Widening that baseline remains future work.
Prove the treatment happened
The preemption null is interpretable because counters showed the starved cache really did preempt 3–5× more.
Separate score from output
“Same accuracy” and “same answer” are different claims. CUDA graphs and MTP can alter text without changing the aggregate score.
Generate and audit claims
Figures are derived from raw results; conclusion blocks name their campaign; independent checks recompute published values.
Boundaries
Where the conclusions
stop.
The results are bounded by the model, runtime, hardware, workloads and configurations tested.
The configuration space for LLM inference is vast. Model format, quantization, KV-cache precision, batching, concurrency, speculative decoding, kernels, memory policies, runtime settings, prompt characteristics and hardware can interact in ways that make exhaustive testing impractical.
For most teams, exploring this space is not the product they are trying to build. Every experiment consumes engineering time, GPU time, and attention—and the search space continues to evolve as models, runtimes, kernels, and hardware change.
RadianVector searches and narrows this space against the application’s workload, then reports the measured trade-offs for deployment.
This study therefore does not attempt to identify a universally optimal configuration. Its conclusions apply to the hardware, versions of the inference stack, workloads and configurations tested here. Other combinations may produce different trade-offs.
Measure these configurations carefully, isolate the effects of specific changes where possible, and provide reproducible evidence that can guide the next experiment.
Prompted reasoning, not native thinking
enable_thinking remained false in this configuration study. Long-form results came from a prompt asking for working and an ANSWER: sentinel. See the thinking-enabled reasoning study →
The tool-call category is excluded
Its prompt forbids prose while the reasoning regime demands working, and it swings sharply between rounds. It cannot fairly rank these arms.
Some 8k arms saw 12k questions
Fifteen items were outside those configurations’ context window. Shared zeros do not bias paired flips, but they depress headline percentages.
Mechanisms remain hypotheses
The experiment identifies relationships and localizes likely components. It did not directly trace token acceptance or prove why longer outputs reduce MTP’s score cost. The MTP follow-up traced the score cost itself to a defect in vLLM's V1 model runner.
RV-380 is a regression suite
It has limited general-knowledge coverage, not broad coverage. Future versions should expand knowledge, safety, multilingual and multimodal evaluation; code execution is reserved for a separate coding benchmark and report.
The deployment envelope is contained here
One model family, a 24 GiB GPU, one vLLM version and a bounded traffic shape. Different model variants, GPU generations, runtimes such as vLLM, SGLang and TensorRT-LLM, and distributed multi-GPU deployments each add new dimensions to the configuration space and multiply the interactions between them.
The takeaway
Inference optimization is not “find the highest tokens per second.” It is finding the fastest configuration that preserves the behavior your application depends on—under the load your users will actually create.