Praecise Engine v0.1.0 Read the docs

Documentation · Method and measurements

What actually speeds up decode on a GB10, ranked

Decode on this hardware runs at ~80% of its 273 GB/s memory-bandwidth ceiling (see gb10-bandwidth-ceiling.md), so the only things that help are the ones that move fewer bytes per token. This is that list, ordered by measured effect, with the things that look promising and are not.

Every figure here is cited to a primary source — an engine PR, vendor documentation, or a paper. Where no primary source exists for a claim, that is stated rather than filled in.

The structural correction that reorders everything

For an MoE model the weight term is active parameters only. On a Qwen3-30B-A3B-class model at 32k context, weights are only ~35% of bytes moved once quantised to 4 bits — so the KV cache, not the weights, dominates.

At short context the reverse holds. This is why the ranking below is context-dependent and why "quantise the weights harder" stops paying so quickly.

1. Pick the architecture first

The largest single difference is decided before any flag is set. Gemma-3-27B (5:1 local/global, sliding window 1024) holds 2.91 GB of KV at 32k; Qwen3-32B holds 8.00 GB. Nothing downstream recovers a 2.7x difference in the dominant term.

2. Quantise the KV cache — the biggest lever, and it is off by default

-ctk q8_0 -ctv q4_0 -fa on

+48% throughput for about 0.002 PPL (llama.cpp PR #7412). Flash attention is required for the quantised-KV path to be taken at all.

This is the highest-value line in this document: a measured, large win, available now, that a default configuration does not give you.

3. Four-bit weights: already solved — do not re-engineer

Use Q4_K_M or MXFP4 and move on. Switching between 4-bit formats is worth about 5%, and one popular choice is worth less than nothing:

Qwen3.5-0.8B, same hardware pp512 tg128 (decode)
NVFP4 25,596 388.12
Q4_K_M 38,339 521.69

Q4_K_M beats NVFP4 at decode by 34%, despite NVFP4 having native Blackwell support and comparable bits per weight (4.50 vs 4.25 — NVFP4 is the heavier format).

The reason, from three independent sources: every engine deliberately bypasses the tensor cores below batch 8. NVIDIA's own TensorRT-LLM routes qinput.shape[0] <= 8 to cuda_scaled_mm, whose NVFP4 small-batch kernel is TILE_M=1 scalar fma() with zero MMA instructions; llama.cpp caps MMVQ_MAX_BATCH_SIZE = 8 and decodes NVFP4 through a 16-entry LUT into dp4a, an sm_61 intrinsic. The one native-FP4 merge reports prefill only — there is no decode row in it anywhere, which is itself the tell.

⚠ Also note Q4_K the format is 4.50 bpw; the widely-quoted ~4.85 is the Q4_K_M recipe, which promotes attn_v and ffn_down to Q6_K. They are not comparable numbers.

⚠ sm_120 FP4 is narrower than "Blackwell has FP4". Per the PTX ISA, .kind::mxf4nvf4 and .kind::mxf4 are supported on sm_120a and sm_121a only — excluded from the family-forward sm_120f path. And tcgen05.mma FP4 kinds are sm_100a/103a/110a only, never sm_120.

4. Speculative decoding — the most underused batch-1 lever

3-4x, lossless, and it composes multiplicatively with everything above. It is a bandwidth amplifier: the weights are read once and several tokens come out.

Note this is about using speculation at all, not about tuning how many tokens to draft — see praecise-method.md for why the tuning question turned out to be second-order.

Things that look promising and are not

ParoQuant is ~10% slower than AWQ (2.1x vs 2.3x), not "near-AWQ speed" as its model card states. It buys accuracy at equal bytes, not speed.

No KV eviction method ships. H2O, SnapKV, Quest and PyramidKV are in neither vLLM nor SGLang. What actually shipped is trained-in sparsity — NSA, DeepSeek DSA — which is a property of the model, not a runtime option. Of the post-hoc methods only Quest works on off-the-shelf dense models (2.23x attention, 7.03x end-to-end), and it is a build-it-yourself item.

Activation sparsity is the wrong lever on fast hardware. TEAL measures 1.53x/1.80x on an A6000 but only 1.25x/1.40x on an A100 — the speedup shrinks as the machine gets faster, because it exploits bandwidth slack that a saturated machine does not have.

Host-side optimisation has near-zero return here. Every published figure is from host-bound H100 serving. GB10 decode at 22.7-38.7 tok/s gives a ~44 ms step budget, against which detokenisation is under 0.01%. Measure whether the workload is CPU-launch-bound before adopting any of it — at batch 1 on this machine it is not.

The honest gap

No primary source anywhere gives a PTQ accuracy table comparing NVFP4, MXFP4, AWQ, GPTQ, Q4_K_M and IQ4_XS on a shared model and benchmark. NVIDIA's only MXFP4 comparison is pretraining (8B, 1T tokens, 36% more tokens needed).

Anyone ranking these formats for post-training quantisation quality is extrapolating — including sources that present such rankings confidently. The throughput numbers above are measured; a quality ranking is not available.

Edit this page on GitHub · Read it as Markdown

    Type to search. Arrow keys move, Enter opens.