Praecise Engine v0.1.0 Read the docs

Documentation · Method and measurements

Acceleration research — what Praecise Engine should implement

Surveyed August 2026 against llama.cpp, vLLM, SGLang and unsloth, with a November 2026 addendum at the end. Every claim below was checked against source, release notes or a paper at the time of writing; version-specific numbers are labelled as such, because several of them have already moved once and will move again.

Praecise Engine is a layer, not a runtime. unsloth is the same shape — it sits above the backends and makes them faster — and is the standard of craft this engine aims at: bit-exact optimizations rather than approximations, gradients and reductions hand-derived so results can be written into buffers that already exist, and honest measurement (they once published a correction because their tokens/sec had been counting startup time). llama.cpp, vLLM and SGLang are backends to integrate and optimize for.

That framing decides what belongs here. An optimization belongs in Praecise Engine when it is expressible above a backend; it belongs in a backend adapter when it needs that backend's kernels or memory layout.

What we already have

  • Speculative decoding: self-speculative block (DFlash) and MTP paths.
  • Per-architecture matmul tuning; multi-arch CUDA binary, Turing -> Blackwell.
  • Architecture coverage including muse-glimmer and qwen4exp.
  • A build-time guard refusing a CUDA build the environment would silently make CPU-only.

Tier 1 — portable, and worth building here

These need no backend's internals, so they are the engine layer's natural work.

1. Suffix / n-gram self-speculation

The cheapest real speedup in the field. A suffix tree over the prompt plus prior generations, frequency-ranked, proposing continuations. No draft model, no extra weights, no GPU memory — pure CPU token-ID matching.

llama.cpp's ngram-mod (shared hash pool, constant memory, variable draft length) documents 0.703 acceptance, against 0.576 for the simpler variant. vLLM ships the same idea as suffix decoding, defaulting to a maximum of 32 speculative tokens chosen adaptively per request per step.

Two properties make this the highest value-per-effort item for a layer: it needs nothing from the backend but token history, and — uniquely — the index can span requests and sessions. Agentic loops, code editing and RL rollouts repeat themselves heavily, and a cross-session suffix index is something no single runtime is positioned to build.

2. Speculation as a portfolio, with honest economics

Speculative decoding is not one feature to switch on. Choose per model, per hardware, per load:

  • Published speedups are almost always quoted at concurrency 1 and decay hard under load. vLLM's P-EAGLE measures 1.55x at c=1 and 1.05x at c=64 on MT-Bench. Speculation trades FLOPs for latency, and a loaded server has none spare.
  • MoE inverts the economics. Parallel verification activates far more experts than single-token decode. llama.cpp measures EAGLE3 at 3.28x on dense Llama-3.1-8B but 1.08x on GPT-OSS-120B.
  • MTP is a latency win that costs throughput: it forces n_parallel=1 and drops prefill to roughly half, because hidden states move device->host.
  • The field has moved past tree verification. DFlash verifies a linear block, DSpark schedules a linear window, and SGLang's CPU/GPU overlap scheduler supports only topk=1. Designing for trees aims at a receding target.

The portable part of the frontier is a scheduling policy: DSpark sizes each verify window from the draft's own calibrated confidence against a pre-profiled steps-per-second curve. That is implementable above any backend that exposes per-position draft confidence. Its paper also flags a trap worth respecting — a scheduler searching without causal early stopping biases the output distribution toward tokens that happen to trigger longer verifications. Causal stopping is load-bearing for correctness, not just speed.

3. Per-hardware x per-quant-type dispatch

The single global MMVQ_MAX_BATCH_SIZE = 8 left up to 49% on the table. llama.cpp replaced it with an empirically swept table: RTX 5090 crosses at batch 5 for Q2_K..Q5_K and 7 for Q6_K; RTX 4090 at 4/6/7; DGX Spark only Q2_K at 6; legacy quants stay at 8.

Two things generalize. The methodology: sweep with clocks pinned (llama-bench -p 1..8 -n 0 -embd 1 -r 50), because thermal drift is larger than the effect being measured. And the classification, which matters more than any single number: MMVQ is bandwidth-bound and wants occupancy; MMQ is register-pressure-bound and wants per-block resources. The same occupancy change measured +3-6% on one and -30% on the other. Never apply one policy globally.

4. Evaluate quantization by trajectory divergence

Perplexity and single-token top-1 both under-detect damage that compounds over a generation. The stronger metric now in use is KL divergence over a multi-token greedy decode against the unquantized reference — unsloth scores 300 held-out examples over 32 tokens. Any quantization decision this engine makes should be justified that way rather than by perplexity.

Per-tensor bit allocation itself is a packaging decision, not a runtime one: llama-quantize --tensor-type takes regexes, so a scheme is expressed as data and costs nothing at inference time.

5. Structured output: pick the backend before dispatch, not after

The mechanism worth adopting is llguidance's lexer/parser split. GBNF has no lexer; llguidance runs a regex lexer in front of the CFG parser, and because there are ~10x fewer lexemes than bytes and tokens tend to align with lexemes, the CFG parser is engaged on under 0.5% of tokens. Measured mask cost is ~50us mean, 0.5ms p99 on a 128k vocab.

Backend selection should be a schema inspection up front, not an exception caught downstream. vLLM discovers xgrammar cannot handle a schema by catching a validation error; the predicate is a pure function and can be run first. This matters beyond tidiness: xgrammar silently drops minLength/maxLength when combined with pattern/format, so output can violate the schema with no error raised.

Grammar failures must degrade, never abort. A grammar that gives up should be reported once, not once per token.

Tier 2 — backend-adapter work

Real wins, but they need a backend's internals, so they belong in adapters.

  • Paged KV. llama.cpp's unified cache preallocates n_ctx * n_seq_max, so concurrency is bounded by maximum context rather than average. A paged prototype reaches 247 concurrent sequences against 25 for unified (2.5x aggregate throughput) at equal low-concurrency performance. Its user-facing defrag knob is already deprecated in favour of context checkpoints.
  • KV cache quantization. FP8 KV measures +14.9% throughput and -14.8% median ITL on Llama-3.1-8B/H100 — but the break-even context length is version-specific (24,889 tokens on one release, 7,010 on a later one; 741,565 vs 22,109 on another model). Two gates: head_dim=256 makes TTFT ~1.6x slower, and on hybrid models skipping sliding-window layers beats quantizing everything.
  • Flash attention is now tri-state and defaults to auto. The trap is that quantized K/V combinations are only compiled with GGML_CUDA_FA_ALL_QUANTS=ON; without it an unsupported combination silently falls off the FA path.
  • Blackwell FP4. The generic NVFP4 MMQ kernel gives +118% to +191% on prefill but is flat (~0.99x) on token generation — a prefill win, not a decode one.

Correctness traps to test for explicitly

Cheap to check, and each one produces plausible-looking bad output rather than an error:

  • BOS handling. Adding or omitting the start-of-sequence token is the single most reported cause of gibberish, endless generation and repetition across engines. Assert it rather than trusting the template.
  • Untrained tokens. Reserved/special tokens ship with garbage embeddings in some base models; constrained decoding can force sampling into exactly those.
  • Per-model numerical quirks — logit scaling, logit softcapping — silently corrupt output when a runtime omits them.
  • Tokenizer bugs live in the runtime, not the artifact. At least one Gemma 4 tokenizer fix was C++-only and needed no GGUF regeneration, so "re-download the model" is often the wrong remedy.
  • A denominator correct per-microbatch is wrong across microbatches of unequal token count. The same class of bug appears in any per-sequence-averaged metric under ragged batching.

What this implies for the roadmap

  1. Suffix/n-gram self-speculation, cross-session — highest value per unit effort, implemented as praecise_runtime::ngram::NgramCache (backend-agnostic; still to wire into the decode loop and measure against the 0.703 reference), and structurally ours rather than any backend's.
  2. A speculation policy layer: pick method and verify-window per model, hardware and load; refuse to speculate where it does not pay (MoE under concurrency).
  3. Per-hardware x per-quant dispatch tables, swept with pinned clocks.
  4. A trajectory-divergence harness, so quantization and speculation claims are measured the way they actually degrade.
  5. Schema-inspecting structured-output routing with a shared compile cache.

Where the frontier converged (verified August 2026)

The models this engine has to serve are no longer a zoo of unrelated designs. Four labs independently shipped the same primitives, which is what makes an acceleration layer tractable at all.

The common design. DeepSeek V4, GLM-5.3-Flash, Kimi K3 and qwen4exp all run MoE with aux-loss-free (noaux_tc) balancing plus shared experts, compressed latent attention, a learned indexer selecting a sparse top-k, and Sinkhorn- projected hyper-connections. The hyper-connection hyperparameters are byte-identical across DeepSeek V4 and GLM-5.3-Flash (hc_mult: 4, hc_sinkhorn_iters: 20, hc_eps: 1e-06), and llama.cpp reuses DeepSeek V4's HC_PRE kernel for Kimi K3. One kernel serves four architectures.

A 3:1 hybrid of linear and full attention is now the mainstream layout -- qwen4exp, GLM-5.3-Flash and Kimi K3 all use it. Gemma 4 (5:1) and muse-glimmer (3:1) alternate sliding and full instead. So the engine needs exactly three attention regimes, not one per model: hybrid linear+full, sliding/full alternation, and indexer-based block-sparse.

This is worth stating plainly about our own work: qwen4exp's hybrid memory, block-sparse QSA indexer and gated hyper-connection residual are the convergent frontier design, not an outlier. That is a reason to invest in the shared primitives rather than treat the architecture as bespoke.

Speculation has four shapes to support: MTP heads in the model, dedicated draft models, DSpark (confidence-scheduled), and DFlash (block diffusion). Note Kimi K3 ships num_nextn_predict_layers: 0 -- no MTP head at all -- and relies on DSpark. An engine that assumes MTP is available will simply have nothing to speculate with.

Architecture traps to test against

Each of these degrades silently rather than erroring:

  • NoPE is spreading. GLM-5.3-Flash and Kimi K3 have no RoPE in the text tower (mla_use_nope: true, qk_rope_head_dim: 0); position comes from the linear-attention recurrence. muse-glimmer drops RoPE on global layers only. Code that assumes RoPE everywhere mis-serves two frontier families.
  • Keep selection machinery out of the quantized path. Every one of these models quantizes routed experts to FP4/MXFP4 while holding indexers, gates and hyper-connection mixers at higher precision. Quantizing an indexer corrupts top-k selection, which degrades quality without any error -- and that applies directly to qwen4exp's QSA indexer.
  • Frontier models increasingly ship no working Jinja chat template. DeepSeek V4 and Kimi K3 have no chat_template at all and need custom encoders; DeepSeeks format uses full-width characters in its role markers. This is the same class of failure as the BOS trap below, and it is why force_pure_content exists as an escape hatch.
  • Some models are not bit-exact between cached and fresh runs (an MXFP4 kernel constraint on DeepSeek V4). Equality assertions in tests will flake; compare scores, not tokens.

Implementation language: what the evidence actually says

Worth recording, because the intuition here is wrong in both directions.

Host overhead is real and large. Profiling vLLM on an H100 with Llama-3-8B found GPU execution was only 38% of step time -- scheduling 29%, API server 33%. An independent study measured scheduling at up to ~50% of end-to-end latency. "The kernels dominate, so the host language does not matter" is false.

But no one solved it by changing language. vLLM stayed in Python and moved input preparation onto the GPU (Model Runner V2, +56% throughput); SGLang's overlap scheduler runs one batch ahead, worth 10-20%; CUDA graphs remove the host from the launch loop entirely. The fix is always taking work off the critical path, not making that work faster.

Two natural experiments already ran, and both cut against a compiled orchestration layer. NVIDIA deleted TensorRT-LLM's C++ backend ("PyTorch is now the sole execution backend", v1.2) for developer efficiency. And TGI -- the Rust-router precedent -- was archived in March 2026, pointing users at vLLM/SGLang/llama.cpp. Rust did not save it; failing to keep pace with model architectures did.

That is the real cost to weigh: EAGLE-3, P-EAGLE, DFlash and DSpark all arrive as Python/Triton PRs against vLLM or SGLang. There is no non-Python adoption path. A compiled layer reimplements each by hand and permanently trails.

Where a compiled layer genuinely wins is embedding. Praecise Engine is a library that loads inside someone else's process, and there Python's single interpreter, GIL and dependency weight are disqualifying -- vLLM's own fix for GIL contention was to move its API server into a separate process. Rust gives a cdylib with no runtime to initialise.

So the split this engine already has is the defensible one: Rust for the embeddable layer, CUDA C++ for kernels. The conclusion to carry forward is not about language at all -- it is that host work must come off the critical path, by preparing inputs on the GPU and overlapping scheduling with compute. Note also that no controlled benchmark of a Rust orchestration layer against a Python one over identical kernels appears to exist publicly; anyone claiming a decisive language win in either direction is extrapolating.

Kernel strategy is set by batch shape, not by hardware

The assumption worth discarding first: "batch-1 decode leaves tensor cores idle, so the vector path always wins." That is true of some engines and false of others, and the difference is not the GPU.

Verified by reading the kernels at HEAD (2026-08-27):

Codebase Batch-1 path
llama.cpp dp4a SIMT GEMV — mmvq.cu contains no MMA at all
FlashInfer scalar FMA + warp shuffle — decode.cuh has none, prefill.cuh has 40
vLLM stays on tensor cores — no GEMV kernel, no dp4a anywhere in csrc/

vLLM's Marlin specializes tiles for small M but keeps the arithmetic on mma.sync.aligned.m16n8k16, adapting through split-K to fill the SMs rather than leaving the tensor cores. Its AllSpark W8A16 path inverts the naive expectation outright: small M runs a tensor-core kernel, while large M (>1024) falls back to cuBLAS.

The dividing line is whether a deployment ever presents M=1 to the GEMM. llama.cpp's steady state is single-user local decode, which is M=1 exactly. vLLM's is a continuous-batching server holding dozens of sequences, so it has little reason to keep a true GEMV path at all.

That is a decision this engine has to make explicitly rather than inherit. A batching server should follow vLLM's precedent — tensor cores plus split-K — even at low nominal batch, because the effective M is the number of concurrent sequences, not one.

Two more measured points worth keeping:

  • MMQ_DP4A_MAX_BATCH_SIZE = 64: even on hardware with FP16 tensor cores, llama.cpp prefers integer SIMT below batch 64.
  • Tensor-core availability shrinks the vector regime rather than widening it (mmvf.cu): Ampere and later use the vector path only at ne11 == 1, Ada up to 4, and hardware without tensor cores up to 8. The per-quant tables carry comments naming the GPU they were tuned on — these are measurements, not derivations, which is the standard any tuning added here should meet.

Two corrections to received wisdom

H100 at batch 1 is launch-bound, not bandwidth-bound — roughly 26% of peak. CUDA graphs are worth ~1.26x there and nothing on an L4. Buying HBM bandwidth does not buy batch-1 latency, so "bandwidth ÷ bytes-per-token" is a ceiling, not a prediction; on a fast GPU at batch 1 the host and launch path bind first. This is the same conclusion the host-overhead numbers reach from the other direction.

Halve NVIDIA's headline tensor figures unless they say dense. The GB200 page states the convention in plain text: the large number is sparse, and dense is half of it.

November 2026: what moved, and what did not

Surveyed 2026-11-13 against llama.cpp 0.7.0 to 0.9.0, vLLM, SGLang and the papers named below. The question was narrow: is there a technique, new since the August survey, that makes decode faster and that this engine can adopt.

No new batch-1 decode kernel was found. Nothing published since August reads fewer bytes per token from a 4-bit model without losing quality. The levers that pay are still the ones this document already ranks: more tokens per pass (speculation), and a model class that emits many tokens per pass (diffusion). The upstream release notes for 0.8.0 and 0.9.0 carry no benchmark figures, so every gain attributed to them below has to be measured here.

Each entry says which of two things it does, because decode here is memory-bandwidth-bound: fewer bytes per token, more tokens per pass, or neither.

# Technique Published gain Lossless Where it lands here Bandwidth effect
1 llama.cpp 0.8.0 and 0.9.0 (ggml 0.28, 0.29): CUDA chunked graph launches, faster low-bit mat-vec, Metal MXFP4 mat-vec, Vulkan split-K on RDNA3/4, Arm SVE Q8_0/Q6_K/MXFP4, SYCL flash attention over a quantised KV cache, the n-gram cache saved to a file None published Yes Linked route; the fork is synced with 0.9.0, built and tested on the CPU only (silicon-coverage.md) Neither; launch cost and kernel efficiency
2 LiLiCorr: a two-layer transformer rescoring a top-K lattice of DFlash drafts Acceptance length +7 to +19% over DFlash at about 3% overhead per block (arXiv 2608.20530, abstract; models and hardware not stated there) ⚠ Not addressed in the abstract The speculation policy layer; needs a trained checkpoint per target model More tokens per pass
3 DFlash drafts in SGLang Spec V2 Over 4.3x baseline and 1.5x MTP on Qwen 3.5 397B-A17B, HumanEval, concurrency 1, 8xB200; the overlap scheduler alone takes Qwen3-8B from about 11.4 to 15.3 ktok/s at concurrency 32 on one B200 Yes, under greedy verification Hosted SGLang route, a configuration change More tokens per pass
4 MoE expert cache on the GPU (--moe-cache-mib), for models that do not fit in VRAM 25.0 to about 40 t/s on an RTX 4090 and 30.8 to 67.8 t/s on an RTX 5090, Qwen3.8-Flash-Next Q4_0 ⚠ Bit-identity not stated The runtime sets the cache size on discrete GPUs Neither; it removes the PCIe path
5 Diffusion language models (DiffusionGemma 26B-A3.8B, Apache 2.0) ⚠ 1,008 tok/s on H100 and 1,288 on H200, FP8, batch 1, in vLLM (trade press; the model card was not reachable) No: a different model, and its maker recommends the autoregressive one for quality Hosted vLLM route. The linked route has only a draft upstream pull request More tokens per pass; the largest such lever
6 ZipServ (ASPLOS 2026): lossless weight compression decoded inside the matrix product Up to 30% smaller weights, 1.22x over vLLM on average at batch 8 and 32, RTX 4090 and L40S Yes, bit-exact Hosted BF16 serving only; a port, not a setting Fewer bytes per token, the only new lossless technique found that does this
7 CUDA graphs kept on for MoE (upstream pull request 26802) 1.13x at 16 sequences, 1.08x at 32, flat at batch 1 (Qwen3-30B-A3B Q4_K_M, RTX 5090, secondary write-up) Yes Already in the fork Neither
8 Arm SME2 matrix products for Q4_K and Q6_K through KleidiAI None in the release notes Yes kleidiai feature; a prefill gain is the expectation Neither

Negative results, recorded so they are not tried twice:

  • The MoE expert cache has made a machine slower. One report on two Radeon R9700 cards under Vulkan fell from 23 to 18.1 t/s. On unified memory (GB10, Apple, Strix Halo) no expert is host-resident, so the cache should have nothing to do; that is an inference from the design, not a measurement.
  • Diffusion decoding is poor off NVIDIA today. The draft upstream pull request reports about 27 tok/s on a Radeon R9700 under Vulkan and no Metal result. llama.cpp 0.9.0 runs the one diffusion model it loads in autoregressive mode only.
  • ZipServ has no batch-1 result, covers BF16 weights only, and its authors report it can lose to cuBLAS on A100 and H800 and to lossy FP8.
  • Native 4-bit floating point is still a prefill feature upstream, not a decode one.
  • Lossy KV eviction and 3-bit KV quantisation were not adopted. The eviction work reports 1.69x over vLLM but gives up accuracy, and it is a preprint read through an aggregator. ⚠ The 3-bit KV reports were not opened.
  • Nothing new for early exit, ternary weights or sub-4-bit weights with a measured 2026 decode figure was found.
  • ⚠ Matrix extensions (AMX, SME, Qualcomm's) for decode were not verified; the one figure seen, about 1.05x at four threads, came from a search snippet. It agrees with the bandwidth ceiling this repository measures.

Two leads not yet read to the source. llama.cpp 0.7.0 lists "MXFP4 MoE with arithmetic decoding and reordering" for SYCL; if that is entropy-coded weights, it is a shipped reduction in bytes per token on Intel, and worth one reading of the pull request. ⚠ OpenVINO GenAI is said to have extended its speculative decoding to CPU, GPU and NPU; the release notes were not opened.

What this changes in the roadmap

  1. The fork is synced with llama.cpp 1.0.0 and built on the CPU only; no GPU backend has been compiled against it. Measure decode before and after on the GB10. Until that measurement exists, no gain is claimed for it.
  2. Let the speculation policy layer choose a DFlash corrector where a checkpoint exists, and refuse where none does.
  3. Size the MoE expert cache from the hardware layer: on for a discrete GPU whose VRAM is smaller than the model, off on unified memory.
  4. Treat a diffusion model as a hosted-route model until the linked route can run one. BatchEngine still refuses them, which is correct.

Sources: llama.cpp release notes for 0.6.0 to 0.9.0 (read through a release mirror); upstream pull requests 29887, 24423 and 26802; arXiv 2608.20530, 2603.17435 and 2606.16332; the vLLM speculative decoding documentation; the SGLang DFlash write-up of 2026-06-15.

Edit this page on GitHub · Read it as Markdown

    Type to search. Arrow keys move, Enter opens.