# Praecise Engine **A backend-agnostic inference-acceleration layer for large language models.** Praecise Engine sits between an application and an inference backend and speeds up token generation — through speculative decoding, hardware-aware tuning, and a uniform serving surface — without the application needing to know which backend is underneath. **llama.cpp is the first supported backend.** It is as agnostic about silicon as it is about backends. NVIDIA, AMD, Intel, Apple, Qualcomm, Arm, Huawei and the rest are each reached by a route that is named, built and tested as far as it can be without the device, and none is the default. [docs/hardware.md](docs/hardware.md) says what is supported on each and how far each claim has been proved; [docs/silicon-coverage.md](docs/silicon-coverage.md) is the plan behind it. --- # Design ``` application / serving layer │ ┌──────────▼───────────┐ │ Praecise Engine │ speculation policy, drafter selection, │ uniform interface │ request shaping, sampling, KV strategy └──────────┬───────────┘ │ pluggable backends ┌─────────────┴───────────────┐ ▼ ▼ llama.cpp vLLM · SGLang · transformers an inference API linked (FFI) TensorRT-LLM · OpenVINO GenAI remote (opt-in) hosted (confined runtime host) │ │ └──────────────┬──────────────┘ ▼ praecise-hardware: which accelerators the machine has, which backend drives each, what a confined engine needs to reach them, and the decode ceiling their memory sets ``` Praecise Engine is the acceleration layer — not a model server on its own, and not a fork of any single engine. Each backend is an adapter. Optimizations that can be expressed generically live in the engine layer; backend-specific work lives in that backend's adapter. --- # What this layer is for, measured Decode is memory-bandwidth-bound. On a DGX Spark (GB10) it already runs at **~80% of the 273 GB/s peak** — the highest fraction of peak of any comparable machine (M4 Max 58%, M3 Ultra 44%). See [docs/gb10-bandwidth-ceiling.md](docs/gb10-bandwidth-ceiling.md). That constrains what a layer above a backend can honestly claim: everything host-side shares the ~20% that is not already saturated. So this engine concentrates on decisions the backend cannot make for itself, and on avoiding losses rather than manufacturing wins. Measured on an RTX 5070 Ti with `qwen3.5-0.8b` and n-gram self-speculation: | prompt | tok/s | drafted | vs baseline | |---|---:|---:|---:| | SQL | 226.3 | 171 | **3.5x** | | code | 94.5 | 73 | **1.5x** | | number list | 63.8 | 0 | — | | JSON | 65.4 | 0 | — | | prose | 62–65 | 0 | — | Speculation is worth 1.5–3.5x on structured text and **nothing** on prose. The decision of *whether* to speculate is where the value is; see [docs/measured-acceptance.md](docs/measured-acceptance.md). --- # Why speculative decoding Single-stream (batch-1) decode is memory-bandwidth-bound: each token streams the model's active weights from memory once, so throughput is bounded by `bandwidth ÷ bytes-per-token`. The software levers are to move fewer bytes per token (quantization, sparsity) or to emit more tokens per weight-read (speculative decoding). Praecise Engine focuses on the latter — techniques that are largely backend-independent, which is why they belong in a layer rather than a fork. For what moves the ceiling itself rather than working within it, see [docs/what-actually-helps.md](docs/what-actually-helps.md). The short version: KV-cache quantization is worth **+48%** and is off by default. --- # What was tried and does not work `praecise_runtime::prax` sized the draft block per position. **Measured on hardware, block size does not change throughput** — a 16x range moves it ~5% non-monotonically, and acceptance is identical to one decimal place at every setting. The module is retired and inert, kept only so the negative result stops the idea being rebuilt. See [docs/praecise-method.md](docs/praecise-method.md). The simulation that predicted otherwise had an oracle bound, seed averaging and 179 passing tests. None of that could detect a mistaken cost model. A simulation is only as right as its assumptions, and the only test of those is measurement. --- # Depending on the engine Depend on the crate you need (today, `llama-cpp-2` / `praecise-runtime`). The backend is pulled as a dependency; a host that already builds the same backend shares it rather than building it twice. `praecise-runtime` defaults to **no backend at all** (`default = []`), so the policy, catalogue and shaping surfaces compile and are testable without pulling llama.cpp or a GPU. --- # Selecting a backend `praecise_runtime::Backend` names the runtimes the engine knows about and reports, per backend, what is actually possible: ```rust use praecise_runtime::Backend; let backend = Backend::parse("vllm")?; // unknown names error, never default backend.ensure_available()?; // refuses rather than silently substituting let caps = backend.supports(); ``` The distinction that matters is **how** a backend is reached, because it decides what acceleration is available at all: | Backend | Integration | Engine-side speculation | Structured output | |---|---|---|---| | llama.cpp | linked (FFI) | yes — the engine drives the decode loop | yes | | vLLM | hosted | no — the runtime owns its decode loop | no | | SGLang | hosted | no | no | | TensorRT-LLM | hosted | no | no | | transformers | hosted (CPU or GPU) | no | no | | OpenVINO GenAI | hosted (Intel CPU, GPU or NPU) | no | no | | an inference API | remote (opt-in) | no — the provider owns the decode loop | no | A *linked* backend is compiled in, so the engine can propose a draft block, verify it, and read per-position logits. A *hosted* backend runs in a runtime host: the engine chooses the model and the sampling parameters, but there is no seam at which to interpose on token-by-token decoding. Speculation against a hosted backend is therefore not merely unimplemented — it is impossible by construction, and those runtimes do their own instead. A *remote* backend runs nowhere on this machine: `praecise-remote` asks an OpenAI-compatible inference API (Groq, Cerebras, SambaNova or any other) and answers in the same types. It is the one route to silicon sold as a service, and the one backend whose prompts leave the machine, so it is opt-in twice: the application has to depend on the crate, and the operator has to name each host in `PRAECISE_REMOTE_ALLOW`. Runtime hosts still have no network. ```rust use praecise_remote::{Allow, Provider, Remote}; let groq = Remote::open(Provider::Groq.endpoint(model), &Allow::from_env())?; let reply = groq.generate(&messages, &config, |text| { print!("{text}"); true })?; ``` An unknown backend name is an error listing the valid ones, never a quiet fallback: a typo that silently ran on a different runtime would invalidate any measurement taken against it, with nothing in the output to show it. --- # Any vendor's silicon A backend is not a vendor. llama.cpp reaches every accelerator a ggml backend is built for; vLLM, SGLang and transformers reach whichever one the packages of their environment drive; TensorRT-LLM is NVIDIA's and OpenVINO is Intel's. The engine asks which is which before it assigns a model: ```rust use praecise_runtime::hardware::Machine; use praecise_runtime::Backend; let machine = Machine::from_env_or_detect()?; // every vendor, read from the OS Backend::Vllm.ensure_reaches(&machine)?; // refuses, naming the backends that do ``` | Silicon | Linked (ggml feature) | ONNX provider | Hosted | |---|---|---|---| | NVIDIA GPUs | `cuda`, `vulkan` | TensorRT, CUDA | vLLM, SGLang, TensorRT-LLM, transformers | | AMD Radeon, Instinct, Ryzen | `rocm`, `vulkan`; `zendnn` on Zen CPUs | MIGraphX; Vitis AI on the NPU | vLLM, SGLang, transformers | | Intel Arc, Xe, Core Ultra NPU, Gaudi | `sycl`, `openvino`, `vulkan` | OpenVINO | vLLM, SGLang, transformers, OpenVINO GenAI | | Apple M-series | `metal` | CoreML | none (hosts need Linux) | | Qualcomm Adreno, Hexagon | `opencl`, `hexagon` | QNN | none | | Arm CPUs, Mali | `kleidiai`, `vulkan` | WebGPU | none | | Huawei Ascend | `cann` | CANN | vLLM, SGLang, transformers | | Google TPU, AWS Trainium, Moore Threads, Cambricon and others | `musa` for Moore Threads | none | vLLM through the vendor's plugin | | Groq, Cerebras, SambaNova | none | none | none; remote, opt-in (`praecise-remote`) | `praecise-hw` prints the accelerators of the machine it runs on, the build features that machine needs, and the route to each device. Only the NVIDIA rows have been run on their silicon; the rest are compiled or wired and wait for a run on the device. The full table, with what each word means, is in [docs/hardware.md](docs/hardware.md). --- # Running several engines An application can serve several models at once, each on the engine that suits it: llama.cpp in its own process, and any hosted backend in a **runtime host**. `praecise_host::Engines` is the one place models are assigned to engines: - **One engine per model.** Every load claims the model id first; a claim held by another engine, or a second host for the same model, is refused with `Error::Conflict`. A model is never in memory twice. - **One memory budget.** Each claim reserves the model's memory from a `Budget` shared by every engine: the application's own ledger, or a `FixedBudget`. - **Confined, not trusted.** A runtime host is a child process the application starts by re-running its own executable as the confinement launcher (call `praecise_host::sandbox::enter_if_requested()` first thing in `main`). It gets new user, network, mount, IPC and PID namespaces, a Landlock ruleset (read-only engine environment and model files, one scratch directory, the GPU device nodes), a seccomp filter and resource limits. It has no network: no interface but an optional loopback with no route out, and no internet socket unless loopback is granted. It speaks to the application only over its standard streams, one JSON object per line, with request ids so an engine that batches sees every outstanding request. - **Pinned environments.** An `EngineEnv` is a directory holding the engine's program, packages and protocol adapter, pinned by the SHA-256 of its whole tree (`praecise_host::engine::digest`). The digest is checked before every start; an environment that changed is refused, never run. - **Supervised.** A host that exits fails the requests it was serving and is restarted in the background from the same verified environment, while every other engine keeps serving. After three failures in a row it is left stopped and reported, until the model is released and served again. - **Accelerator or refusal.** On a machine with an accelerator of any vendor, an engine that cannot use it refuses to load rather than falling back to the CPU. A host is given the device nodes of every vendor (NVIDIA, AMD, the kernel's DRM and compute-accelerator classes, Ascend, Neuron and the rest), the vendor runtime directories, and the device-selection variables the operator set. The hosted backends are driven through their offline Python engine APIs (vLLM `AsyncLLM`, SGLang `Engine`, TensorRT-LLM `LLM`, transformers, OpenVINO GenAI `LLMPipeline`) by one adapter, `praecise-host/adapters/praecise_adapter.py`, which `engine::install_adapter` writes into a provisioned environment before it is pinned. No engine opens an HTTP server. `praecise_host::generation` maps `GenerationConfig` and `InferenceResult` onto the protocol, so a hosted engine answers with the same types, token counts and stop reasons as the linked one; a reply without token counts is refused rather than metered as free. A kernel that forbids unprivileged user namespaces (for example `kernel.apparmor_restrict_unprivileged_userns=1`) refuses the confinement, and the error names the setting; grant the application's executable `userns` in its AppArmor profile rather than lowering the setting for the whole machine. On Ubuntu that is one profile per executable, for example `/etc/apparmor.d/praecise-job`: ``` abi , include profile praecise-job /opt/praecise/bin/praecise-job flags=(unconfined) { userns, } ``` followed by `apparmor_parser -r /etc/apparmor.d/praecise-job`. Where that is not possible an operator may choose `--confinement landlock-only` explicitly: the same Landlock ruleset and syscall filter (no internet sockets, no file outside the rules), without the namespaces. It is never chosen on its own. ## Job engines Beside the model engines, `praecise_host::job` runs job engines: a fixed program with typed parameters, input files in, output files and progress out, run once to completion. A job engine is a pinned environment (the same SHA-256 tree digest, checked before every run) and one Python adapter, `praecise-host/adapters/praecise_job.py`, compiled into the binary and placed read-only beside the run's scratch directory. Every run is confined like a model host: no network, the environment, model caches and inputs read-only, one scratch directory writable, the GPU device nodes; it is supervised (time limit, the whole process tree stopped at the end, the last lines of its error output on failure); on a machine with a GPU a program refuses rather than run on the CPU. Parameters are checked against each program's declaration (type and range) before anything starts, and again by the program. The programs drive NVIDIA's open simulation systems. They are NVIDIA's own products and run on CUDA; the ones built on torch also run on AMD ROCm, which presents as `cuda`: | Program | System | Out | |---|---|---| | `sionna.radio_map` | Sionna RT | radio_map.npz, radio_map.png | | `warp.example` | Warp | the example's USD stage | | `newton.example` | Newton | sim.usd | | `cosmos.generate` | Cosmos 3 (guardrail forced on) | video.mp4 | | `physicsnemo.darcy_fno` | PhysicsNeMo | fno_darcy.pt and its error on unseen fields | | `cudaq.ghz`, `cudaq.maxcut` | CUDA-Q | sampled states, the best cut against the optimum | | `boltz.predict` | Boltz-2 | structures, confidence, affinity | Each writes result.json. The `praecise-job` binary runs one: ``` praecise-job programs # the catalogue, with every parameter's type and range praecise-job digest /opt/envs/cudaq # the digest to pin an environment by praecise-job run --engines engines.json --program cudaq.ghz --param qubits=24 --out out/ ``` `engines.json` names each engine's environment and pin: `{"engines": {"cudaq": {"root": "/opt/envs/cudaq", "digest": "", "read_only": [...], "env": {...}}}}`. Progress is printed as JSON lines and the last line is the result. --- # Perception Video in, per-frame object tracks out, as an open replacement for a Metropolis/DeepStream perception pipeline. `praecise-codec` decodes the video in process, an ONNX detector from `praecise-onnx` (RF-DETR Nano to Large, Apache-2.0, or D-FINE) runs on every frame on the GPU's execution provider, and ByteTrack (an independent Rust implementation: Kalman prediction, two-stage association with low-score detections, minimum-cost assignment) gives every person and vehicle a stable id, box, score and velocity. With an image-to-ground homography each track also has a ground position and speed in metres. Frames can be written as NVIDIA Metropolis `nv.Frame` protobuf messages, the input Halos Safety Core reads. ``` praecise-track --model rfdetr-medium.onnx --input-size 576 --video in.mp4 --out out/ --nvframe ``` writes `tracks.json`, `summary.json` (frames per second, time per stage, peak memory) and `frames.nvpb` (one `nv.Frame` per frame, each prefixed by its length as a varint). --- # What the layer provides ## Speculation policy `praecise_runtime::spec_policy` decides *whether* to speculate given model shape and load, and returns a `reason` with every plan — a policy that cannot explain itself gets switched off. It refuses where speculation does not pay: MoE under concurrency, a loaded engine, a request too short to amortise a draft. ## Self-speculation with no draft model `praecise_runtime::ngram` drafts from token history alone — no second model, no GPU memory. It is deliberately **not tied to a request**, so an agent loop or a code-editing session reuses patterns across turns, which no single-request runtime is positioned to do. This is the technique behind the 3.5x above. ## Drafter catalogue with licence enforcement `praecise_runtime::drafters` records published drafter checkpoints **with their licences**, which are frequently not the licence of the model they draft for. `Qwen3.8-27B-DFlash2` is Apache-2.0; `GLM-5.3-Flash-DFlash2` is CC-BY-NC-ND. Both sit on the same hub and a checkpoint path is just a string. Only permissive checkpoints are offered. The rest are kept in the catalogue so that naming one explains *why* it was refused, and a checkpoint whose card states no licence at all is refused too — absent terms grant nothing. It also preflights four documented ways a drafter deployment silently underperforms: a block size the checkpoint was not trained at, a model runner that quietly downgrades a newer drafter to an older one, grammar conflicts, and partial-vocabulary sampling. Each produces a smaller speedup rather than an error, which is why they are worth checking for. ## Request shaping `praecise_runtime::shaping` makes a batch cheap before it reaches a backend. The sharpest item: `all_greedy` and `no_penalties` are **batch-wide booleans**, so one request using a penalty drags the entire batch onto the expensive sampling path — including every request that asked for none. Penalties cost 6–47 ms/step at batch 4; the sampler itself costs ~43 µs. See [docs/request-shaping.md](docs/request-shaping.md). --- # Crates - `llama-cpp-sys-2` — low-level FFI; pulls the llama.cpp backend (the Praecise fork, synced with upstream llama.cpp 1.0.0 of 2026-11-13; that sync is built on the CPU only, and no GPU backend has been compiled against it). - `llama-cpp-2` — safe Rust bindings, including the speculative-decode primitives. - `praecise-hardware` — the hardware layer: which accelerators a machine has (every vendor, read from the operating system), which backend drives each, what a confined engine needs to reach them, the memory and bandwidth of the systems worth planning for, and the `praecise-hw` binary. No dependencies. - `praecise-runtime` — the backend-agnostic acceleration runtime and inference API. - `praecise-host` — runtime hosts and the engine set: confined, supervised engines, one per model, within one memory budget; job engines and the `praecise-job` binary. - `praecise-remote` — generation on an inference API outside the machine (Groq, Cerebras, SambaNova or any OpenAI-compatible endpoint), in the runtime's types. Opt-in: nothing opens a connection unless the application depends on this crate and the operator permits the host. - `praecise-ffi` — the stable C ABI over the runtime's backend-agnostic types (config, result, stop reason, error), for callers in other languages. - `praecise-diffusion` — flow-matching image, video, audio and 3D generation on the same ggml backend, device and memory budget as the language-model family. - `praecise-provenance` — signed C2PA content credentials embedded in generated PNG, WAV and MP4 files, and their verification. - `praecise-train` — the training substrate: the deterministic replayable step, the hash-linked step log, content-hashed checkpoints with lineage, objectives, optimizers and parameter-delta primitives. - `praecise-codec` — in-process video coding: H.264, H.265 and AV1 decoding from MP4 and Matroska/WebM, H.264 and AV1 encoding to MP4 (AV1 also to WebM) with an Opus soundtrack. Generated videos are written as MP4. - `praecise-perception` — video in, object tracks out: detection on every decoded frame, ByteTrack, `nv.Frame` output; the `praecise-track` binary. - `praecise-onnx` — the ONNX model families: speech-to-text, object detection, promptable and text-prompted segmentation, image encoders, time-series forecasting and text embedding. Sessions register the execution provider the machine's GPU needs (CUDA and TensorRT, MIGraphX, OpenVINO, QNN, CANN, CoreML, or DirectML and WebGPU on any vendor's GPU) and refuse a model they cannot accelerate rather than running it on the CPU. A neural processor is used when the operator asks for it. - `praecise-moe` — expert-parallel mixture-of-experts execution: expert FFNs and gating from safetensors blobs, GGUF block-quantized expert weights, a memory and disk residency cache, and the combine step, on the CPU (AVX-512-VNNI, AVX-VNNI, the Arm dot-product extension), CUDA or any other GPU through wgpu. The `cuda` feature builds for the CUDA toolkit that nvcc reports (12 or 13); `cuda-12` and `cuda-13` pin the version on a build host without a toolkit. --- # Testing ``` gcloud builds submit --config cloudbuild-test.yaml --project . ``` Runs fmt, clippy and the test suite on the hardware layer and the backend-agnostic runtime, plus a compile check of the backend-gated code. It validates *correctness* on x86 — not an aarch64 artifact, and nothing about CUDA kernels or GB10-specific tuning, which need real hardware. The `silicon-coverage` workflow covers what can be proved about hardware without the hardware, on x86-64 and AArch64 Linux, macOS and Windows: the hardware layer and the decisions made from it, every ONNX execution provider compiled against ONNX Runtime, and the ggml builds that need no vendor SDK (ZenDNN, KleidiAI, Metal) linked into a binary. It proves that a route builds, not that a device runs. --- # Hardware support Praecise Engine runs on any vendor's silicon, not one. This page is the record of what is supported on each, by which route, and how far each claim has been proved. The plan behind it is [silicon-coverage.md](silicon-coverage.md); the reasoning about memory, packaging and interconnect that shapes it is [foundations.md](foundations.md). ## How far a claim goes Every row carries one of four words. They are not interchangeable. | Word | Meaning | |---|---| | **verified** | Inference has run on that silicon, including speculative decode. | | **built** | Compiled and linked into a binary. Nothing has run on the device. | | **compiled** | Type-checked against the vendor's API. It is not linked against the vendor's runtime. | | **wired** | The feature exists and sets the upstream switch. It has not been built here, usually because it needs the vendor's SDK. | Only NVIDIA rows are verified today. Every other row is a route that exists and is tested as far as it can be without the hardware; it becomes verified when someone runs it on the device and records the run here. The `silicon-coverage` workflow builds the routes that need no vendor SDK (ZenDNN, KleidiAI, Metal, the AVX-512 options) and compiles every ONNX provider on each change. A row moves from `wired` to `built` when that workflow has built it; until its first run the rows it covers stay `wired` ([silicon-coverage.md](silicon-coverage.md)). The default CPU build and the crates above it have been built once by hand, on Windows x86-64. ## What a machine has `praecise-hardware` reads the accelerators of a machine from the operating system: PCI vendor and class codes, the kernel driver bound to each DRM and compute-accelerator device, and the device nodes vendor drivers create. It calls no vendor library, so it works where a vendor's tools are absent. ``` praecise-hw # accelerators, routes, build features, CPU extensions, device nodes praecise-hw catalogue # decode ceiling of each system in the catalogue praecise-hw measure # memory bandwidth this machine's CPU threads reach ``` `PRAECISE_ACCELERATORS` declares the machine instead of detecting it (`amd:gpu,amd:npu`, `google:accelerator`, `none`). It is the only way to name an accelerator the system shows no stable sign of, a cloud TPU for instance. Detection is tested against described machines on every build host: a multi-GPU server with a baseboard VGA device, Ryzen AI and Core Ultra laptops with their NPUs, Gaudi, Ascend and Cambricon cards, a container with only device nodes, Snapdragon and Mali system-on-chips, IBM Z. On Windows the display-adapter and compute-accelerator device classes are read from the registry; the display class is checked on a real machine, the compute-accelerator class on none. ## Three routes onto an accelerator 1. **Linked**: a ggml backend of the bundled llama.cpp. The engine drives the decode loop, so speculation and structured output are available. 2. **ONNX**: an ONNX Runtime execution provider, for the ONNX model families (speech, detection, segmentation, embedding, forecasting). 3. **Hosted**: vLLM, SGLang, transformers, TensorRT-LLM or OpenVINO GenAI in a confined runtime host. The runtime owns its decode loop. A fourth route leaves the machine. **Remote** (`praecise-remote`) asks an inference API for the reply. It is the only way to silicon sold as a service, and it is off unless the application is built with it and the operator names each host prompts may go to (`PRAECISE_REMOTE_ALLOW`). `praecise_runtime::Backend::reaches` answers, per accelerator, whether a backend has a route onto it, and `ensure_reaches` refuses a backend with none, naming the ones that have. ## Coverage by vendor Feature names are the same on `praecise-runtime`, `praecise-diffusion`, `llama-cpp-2` and `llama-cpp-sys-2`. ONNX feature names are those of `praecise-onnx`. | Vendor | Silicon | Linked (ggml feature) | ONNX provider (feature) | Hosted | |---|---|---|---|---| | NVIDIA | GeForce, RTX, data-centre GPUs, GB10, Jetson | `cuda` verified; `vulkan` wired | TensorRT, CUDA (`cuda`) compiled | vLLM, SGLang, TensorRT-LLM, transformers | | AMD | Radeon, Instinct, Ryzen integrated GPUs | `rocm` wired; `vulkan` wired | MIGraphX (`rocm`) compiled | vLLM (ROCm), SGLang, transformers | | AMD | XDNA NPU | none | Vitis AI (`vitis`) compiled | none | | AMD | Zen CPUs | `zendnn` wired | CPU provider | every hosted engine on its CPU path | | Intel | Arc, integrated Xe, data-centre GPUs | `sycl` wired; `openvino` wired; `vulkan` wired | OpenVINO (`openvino`) compiled | vLLM (XPU), SGLang, transformers, OpenVINO GenAI | | Intel | Core Ultra NPU | `openvino` wired | OpenVINO on the NPU (`openvino`) compiled | OpenVINO GenAI | | Intel | Gaudi | none | none | vLLM (Gaudi plugin), SGLang, transformers | | Intel | Xeon | AVX-512 and AMX options wired | CPU provider | every hosted engine on its CPU path | | Apple | M-series GPU | `metal` wired | CoreML (`coreml`) compiled | none: runtime hosts need Linux | | Apple | Neural Engine | none | CoreML | none | | Qualcomm | Adreno GPU | `opencl` wired; `vulkan` wired | QNN on the GPU (`qnn`) compiled | none | | Qualcomm | Hexagon NPU | `hexagon` wired | QNN on HTP (`qnn`) compiled | none | | Arm | Mali, Immortalis | `vulkan` wired | WebGPU (`webgpu`) compiled | none | | Arm | AArch64 CPUs | `kleidiai` wired; NEON, dot-product, int8-matmul, SVE, SME selected at run time | CPU provider | every hosted engine on its CPU path | | Huawei | Ascend NPU | `cann` wired | CANN (`cann`) compiled | vLLM (vllm-ascend), SGLang, transformers (torch_npu) | | Google | TPU | none | none | vLLM (tpu-inference) | | Amazon | Trainium, Inferentia | none | none | vLLM (vllm-neuron) | | Moore Threads | MTT GPUs | `musa` wired; `vulkan` wired | none | vLLM (vllm-musa), SGLang, transformers (torch_musa) | | Cambricon | MLU | none | none | vLLM (vllm-mlu) | | MetaX, Iluvatar, Enflame | GPUs and GCUs | `vulkan` wired where the driver has it | none | vLLM (vendor plugin) | | Tenstorrent, FuriosaAI | Blackhole, RNGD | none | none | the vendor's vLLM-compatible server | | IBM | Z on-chip accelerator | `zdnn` wired | none | vLLM (vllm-spyre, for Spyre) | | Esperanto | ET-SoC | `et` wired | none | none | | RISC-V | RVV, SpacemiT | CPU backend; RVV options default on | CPU provider | none | | LoongArch | LSX, LASX | CPU backend; LSX and LASX default on | CPU provider | none | | Groq, Cerebras, SambaNova | LPU, wafer-scale engine, RDU | none | none | none; remote only (`praecise-remote`, opt-in), compiled and tested against a local server, not against the provider | | any GPU | DirectX 12 or WebGPU | `webgpu` wired | DirectML (`directml`), WebGPU (`webgpu`) compiled | none | Notes that qualify the table: - **ROCm.** ONNX Runtime removed its ROCm execution provider in 1.23. The `rocm` feature of `praecise-onnx` builds MIGraphX, the provider that runs on ROCm. On Ryzen integrated GPUs Vulkan is often level with or ahead of HIP on decode; benchmark both before choosing. - **Neural processors.** An NPU takes quantized, static-shape models, so an ONNX session goes to one only when the operator asks for it (`OnnxConfig.npu`). It is registered ahead of the GPU's provider. - **Portable providers.** DirectML (Windows) and WebGPU serve a GPU whose own feature is not built. A build with neither, and without the vendor's feature, refuses ONNX models on a machine with a GPU rather than running them on the CPU. - **Vendor SDK backends.** `sycl`, `openvino`, `cann`, `musa`, `zdnn`, `webgpu`, `hexagon`, `et` and `opencl` link a vendor runtime this build script does not name. A static build leaves that link to the final crate, as `opencl` always has; a `dynamic-link` build carries it in the shared libraries. `cuda`, `rocm`, `mkl` and `zendnn` are linked by the build script; KleidiAI's archive is installed beside the ggml ones and linked with them. - **Hosted engines need Linux.** Confinement is namespaces, Landlock and seccomp. On macOS and Windows only the linked and ONNX routes exist. - **API-only silicon.** Groq, Cerebras and SambaNova are reached over a network API and no other way. A runtime host has no network by design, so they are reached through `praecise-remote` and not through a host: a separate crate, with an allow-list of hosts and no default. The provider owns the decode loop and the hardware, so there is no speculation, no logit access and nothing to measure but the reply. The hyperscalers' internal chips have no public interface at all. ## NVIDIA CUDA (llama.cpp backend) The CUDA build is a **multi-architecture** binary: one build runs across the NVIDIA GPU generations below. Architecture-specific fast paths are compiled in and selected at runtime, so a single artifact serves a mixed fleet. | Arch (SM) | Generation | Support | Notes | |---|---|---|---| | 75 | Turing (RTX 20xx, T4) | supported | int8 tensor cores | | 80 | Ampere (A100) | supported | async loads, faster TC | | 86 | Ampere (RTX 30xx) | supported | | | 89 | Ada (RTX 40xx, L4/L40) | supported | | | 90 | Hopper (H100/H200) | supported | | | 120a | Blackwell (RTX 50xx, B-series) | supported | FP4 tensor cores (CUDA ≥ 12.8) | | 121a | Blackwell (Grace-Blackwell) | supported + **verified** | arch-gated MMQ/MMA tuning; speculative decode validated here | - **Toolkit:** CUDA ≥ 12.8 for the Blackwell (120/121) paths; earlier toolkits build the ≤ 90 set. Older virtual archs (50/61/70) are included on CUDA < 13. - **Tuning specifics** (per-arch MMQ config, MMA/flash-attention paths) are guarded by capability macros and only activate on the architecture they target. Adding a specialization never regresses the others. - **Verified** means we have run inference (including speculative decode) on that architecture. Other rows are supported by the backend's standard architecture coverage; we mark them verified as we test them. ## CPUs The CPU is an inference device on every machine and the only one on many. | Architecture | Selected at build | Selected at run time | |---|---|---| | x86-64 | AVX, AVX2, FMA, F16C, AVX-VNNI, AVX-512 (F, CD, VL, DQ, BW together), AVX-512 VBMI/VNNI/BF16, AMX tile/int8/bf16, from the Rust target features | MoE expert kernels: AVX-512-VNNI, then AVX-VNNI, then scalar | | AArch64 | `GGML_CPU_ARM_ARCH`; KleidiAI (`kleidiai`) | KleidiAI picks SVE, SVE2, SME, SME2 kernels; MoE expert kernels: NEON dot-product, then NEON | | RISC-V | RVV, Zfh, Zvfh, Zicbop, Zihintpause on by default | | | LoongArch | LSX, LASX on by default | | | s390x | VXE with `GGML_NATIVE`; `zdnn` | | `praecise-hw` prints the extensions of the CPU it runs on and the ggml options they enable. ## Confined engines reach every vendor's device A runtime host is given exactly the device nodes that exist: NVIDIA's, AMD's `kfd`, the kernel's DRM and compute-accelerator classes (`/dev/dri`, `/dev/accel`: Intel and AMD NPUs, Gaudi, Qualcomm Cloud AI), Ascend's `davinci*`, Neuron's, Tenstorrent's, Adreno's and Hexagon's, Mali's and Cambricon's. Vendor runtime directories outside the system library paths (`/opt/rocm`, `/opt/intel`, `/opt/habanalabs`, `/opt/aws/neuron`, …) are readable, and the device-selection variables the operator set (`CUDA_VISIBLE_DEVICES`, `HIP_VISIBLE_DEVICES`, `ZE_AFFINITY_MASK`, `ASCEND_RT_VISIBLE_DEVICES`, …) are carried into the host. Inside the host, the adapter takes whichever accelerator the environment's packages drive: torch's `cuda` (NVIDIA and ROCm), `xpu`, `mps`, `hpu`, `npu`, `musa`, or the platform vLLM reports, which includes every vendor plugin. A machine with an accelerator the engine cannot use refuses the load. Each vendor's vLLM plugin pins the upstream release it was built against, so each needs an engine environment of its own. ## Adding an architecture or backend 1. Confirm it builds and runs a correct forward pass on real hardware. 2. Gate any specialization behind a capability check so other targets are unaffected. 3. Add its vendor, detection sign and route to `praecise-hardware`, with a test against a described machine. 4. Record it here, marking `verified` only after a real run. --- # Silicon coverage Praecise Engine is described as backend-agnostic. Until November 2026 that was true of its backends and not of its hardware: every route that touched a device assumed NVIDIA, with AMD and Apple as partial exceptions. This page is the account of that difference: what the engine had, every gap found, what was done about each, and what remains that cannot be done without the hardware. Surveyed 2026-11-12 against vendor documentation, release notes and source. Version-specific facts are dated, because they move. ## Principles 1. **No vendor is the default.** A machine's accelerators are read from the operating system, by vendor and kind. Nothing above that layer carries a private idea of what "a GPU" is. 2. **An accelerator is used or the load is refused.** The rule that already held for NVIDIA holds for every vendor: a machine with an accelerator never runs a model on the CPU because the build lacks a backend. The refusal names the build that would work. 3. **A claim says how far it goes.** `verified`, `built`, `compiled` and `wired` are different words ([hardware.md](hardware.md)). A route that was never run on its silicon is not called supported without that qualifier. 4. **Published figures rank; measurements decide.** The catalogue's bandwidth figures are ceilings. Decode on a measured machine reaches 44% to 80% of them ([gb10-bandwidth-ceiling.md](gb10-bandwidth-ceiling.md)). 5. **Decode is memory-bound on every vendor.** The arithmetic that places a GB10 places an Instinct card, an M-series Mac and a TPU. Hardware is ranked by memory and its bandwidth first, compute second. ## What the engine had | Area | State before | |---|---| | ggml features on `llama-cpp-sys-2` | cuda, metal, vulkan, opencl, rocm, sycl, openvino, musa, cann, webgpu, blas, mkl, zdnn, rpc | | ggml backends in the fork with no feature | Hexagon, ZenDNN, ET, VirtGPU, KleidiAI | | Features reachable from `praecise-runtime` | cuda, rocm, metal, vulkan | | Features on `praecise-diffusion` | cuda, metal | | ONNX providers | CUDA and TensorRT, ROCm and MIGraphX, CoreML; `Gpus { nvidia, amd }` | | MoE expert kernels | CPU (AVX-512-VNNI, NEON), cuBLAS, wgpu | | Hosted backends | vLLM, SGLang, TensorRT-LLM, transformers; the adapter required `torch.cuda` | | Host confinement | device nodes `nvidia*`, `kfd`, `/dev/dri`; `CUDA_VISIBLE_DEVICES` and `NVIDIA_VISIBLE_DEVICES` carried through | | Hardware detection | none in the engine; two crates each checked `/dev/nvidiactl` and `/dev/kfd` themselves | | Training kernel classes | CPU, CUDA, Metal, Vulkan | | Documentation | NVIDIA architectures; everything else "planned" | | Verification | GB10 (sm_121) and RTX 5070 Ti | ## Gaps, and what was done ### Closed | Gap | What exists now | |---|---| | No single account of a machine's accelerators | `praecise-hardware`: detection for 21 vendors from PCI ids, kernel driver bindings and device nodes; Linux, Windows and macOS; `PRAECISE_ACCELERATORS` to declare a machine; tested against described machines | | No account of which software drives which silicon | `praecise_hardware::route`: ggml backend, torch device and vLLM platform per accelerator; `praecise-hw` prints the build a machine needs | | A backend could be assigned to silicon it cannot reach | `Backend::reaches`, `reachable` and `ensure_reaches`; a refusal names the backends that do reach the machine | | Intel silicon had no hosted route of its own | `Backend::OpenVino`: OpenVINO GenAI on Intel CPUs, GPUs and NPUs, as a hosted engine | | Hosted engines ran only on CUDA | The adapter takes the accelerator the environment drives: torch `cuda` (NVIDIA, ROCm), `xpu`, `mps`, `hpu`, `npu`, `musa`; vLLM's own platform, including every vendor plugin; SGLang's device argument. TensorRT-LLM tells ROCm from CUDA and refuses it | | The memory cap was read from `torch.cuda` | Read from the device that is present; an engine whose platform reports no size takes `memory_fraction` from the operator or refuses | | A confined host could open only NVIDIA and AMD devices | Device nodes of every vendor; vendor runtime directories; every vendor's device-selection variables | | ONNX models ran on the CPU on Intel, Qualcomm and Huawei machines | OpenVINO, QNN, CANN, Vitis AI, DirectML and WebGPU providers, with the same refuse-or-accelerate planning; NPUs on request | | The ROCm execution provider no longer exists in ONNX Runtime (removed in 1.23) | The `rocm` feature builds MIGraphX; the dead provider is gone | | Five ggml backends in the fork had no feature | `hexagon`, `zendnn`, `et`, `virtgpu`, `kleidiai` | | Ten ggml backends could not be selected from `praecise-runtime` | Every backend is a feature of `praecise-runtime` and `praecise-diffusion` | | AVX-512 foundation and AMX were never passed to ggml | Mapped from the Rust target features; AVX-512 only when all five subsets are present | | A published `llama-cpp-sys-2` crate dropped the sources of ten backends | The package include list names every backend directory | | MoE kernels ignored AVX-VNNI and the Arm dot-product extension | AVX-VNNI tier for Intel client cores; `SDOT` tier for Armv8.2 and later; tiers proved equal bit for bit where the CPU has both | | Diffusion refused the CPU only on NVIDIA, AMD and Apple machines | The refusal covers every accelerator a ggml backend can drive | | Training recognised four device families | Every backend's devices are recognised and a run is held to its device; deterministic mode refuses them until audited | | `praecise-track` detected NVIDIA and AMD itself | Reads the machine from the hardware layer | | No test ran on ARM, macOS or Windows for the hardware paths | `silicon-coverage` workflow: four host platforms, every ONNX provider, and ZenDNN, KleidiAI and Metal linked into a binary. The workflow has not run yet (see the open gaps below) | | No figure for non-NVIDIA systems | `praecise_hardware::silicon`: memory and bandwidth of 17 systems across six vendors, each tagged with its source and whether it ships; `praecise-hw catalogue` | | Bandwidth was inferred from a part's name | `praecise-hw measure` measures what the CPU threads of the machine reach | | Silicon sold only as a service had no route at all | `praecise-remote`: an opt-in client for OpenAI-compatible inference APIs (Groq, Cerebras, SambaNova or any other), answering in the runtime's types. It exists only in a build that depends on it and sends only to hosts the operator names in `PRAECISE_REMOTE_ALLOW` | | bindgen did not compile from the workspace lockfile, so nothing that links llama.cpp built | bindgen is built without its `prettyplease` feature (bindgen 0.73 parses with syn 2; prettyplease 0.3 moved to syn 3) | ### Open: needs the hardware Each of these is a real gap that cannot be closed from a build host. None is hidden behind a passing check. | Gap | What closing it takes | |---|---| | Every non-NVIDIA row is unverified | A run on the silicon, recorded in [hardware.md](hardware.md). In rough order of reach: AMD Instinct and Radeon (HIP, Vulkan), Apple M-series (Metal), Intel Arc (SYCL, OpenVINO), Ryzen AI and Core Ultra NPUs, Ascend, Snapdragon | | Per-hardware, per-quantization dispatch tables exist only for NVIDIA | Pinned-clock sweeps on each architecture (`gfx942`, `gfx950`, `gfx1151`, Apple, Arc), as [acceleration-research.md](acceleration-research.md) describes for NVIDIA | | No deterministic kernel set off CPU and CUDA | An audit of reduction order per backend on its own device. HIP and MUSA compile the CUDA kernel sources but use hipBLAS and muBLAS and split by a different device, so the CUDA audit does not carry over | | The fork's sync with llama.cpp 1.0.0 is proved on the CPU only | The fork is synced with upstream 1.0.0 (ggml 1.0.0), which removes the deprecated API, moves the multimodal code out of `clip` and changes the CUDA attention and quantized-product kernels the fork patches. Built on an x86-64 CPU: the merged tree, with upstream's architecture test (155 of 155) and gradient test passing; `llama-cpp-sys-2` and `llama-cpp-2` with `mtmd` type-check against it. Nothing has been compiled on any GPU backend, the other crates have not been compiled against it, and the workspace's own tests have not run. Closing this takes the remaining crates type-checked and the runtime tests run, then a CUDA build, the backend comparison tests on each backend, and the fork's patches re-validated on hardware | | No hosted engines on macOS | Confinement for macOS written and tested on a Mac. Until then Apple is reached through linked Metal and CoreML, and MLX and vllm-metal are not hosted | | Static links for vendor-SDK backends are unproven | A build of `sycl`, `openvino`, `cann`, `musa`, `zdnn`, `webgpu`, `hexagon` and `et` on a machine with each SDK. Until then a static build leaves the vendor runtime to the final crate and a `dynamic-link` build carries it | | The Windows compute-accelerator device class is unchecked on an NPU | A run of `praecise-hw` on a Windows machine with an NPU | | Cambricon's torch device is unconfirmed from a primary source | Its `torch_mlu` package; until then Cambricon is reached through its vLLM plugin only | | The `silicon-coverage` workflow has not run | Its first green run. The CPU build of llama.cpp and every crate above it were built and type-checked once, on Windows x86-64 with MinGW; nothing was built on Linux, AArch64 or macOS. Until the workflow runs, `zendnn`, `kleidiai`, `metal` and the AVX-512 options are `wired`, not `built`, and the ONNX providers are compiled on that one build host only | | The remote backend has not met a real provider | A run against Groq, Cerebras and SambaNova with a key. It is tested against a local server speaking the same protocol; whether each provider returns usage in a stream, and which length field each takes, is read from their client libraries and not from a response | | Quantum jobs simulate on NVIDIA GPUs or the CPU only, and no quantum processor is reachable | A simulation program on a simulator with an AMD path (Qiskit Aer on ROCm, `lightning.kokkos` or QuEST with HIP), run on that hardware; and, for a real device, an opted-in network route for jobs. See "Quantum" below | ### Out of scope, by design - **Silicon with no public interface.** Microsoft Maia, Meta MTIA and OpenAI's own chips are not reachable by a third party at all. - **A network inside a runtime host.** Hosts and jobs stay without one. The remote backend is not a host: it is a separate crate, off unless built in, and it sends only where the operator permits. - **Job engines.** Warp, Sionna, Cosmos, PhysicsNeMo and CUDA-Q are NVIDIA's own systems. The ones built on torch run on ROCm, which presents as `cuda`; the rest are CUDA programs. An equivalent on another vendor's stack is a different program, not a port. ## The state of silicon, November 2026 What the list above was built against. ⚠ marks a figure from press coverage that was not confirmed against the vendor. **AMD.** Instinct MI350X/MI355X (CDNA 4, 288 GB HBM3E, ⚠ 8 TB/s) ship with native MXFP8/MXFP6/MXFP4; the MI400 series launched 2026-07-23 with MI455X (⚠ 432 GB HBM4, 19.6 TB/s) in the 72-GPU Helios rack. ROCm is 10.1.0 (2026-10-05), built by TheRock; it pairs with vLLM 0.29 and SGLang 0.5.18. Ryzen AI Max (Strix Halo) carries 128 GB unified at ⚠ 256 GB/s. ONNX Runtime dropped its ROCm provider in 1.23; MIGraphX is now a plugin provider. **Intel.** Arc Pro B-series (to 32 GB) are the current GPUs; Crescent Island (Xe3P, 160 GB) is enabled in oneAPI 2026.2 and not shipping. Gaudi 3 is supported through a vLLM hardware plugin; the vLLM fork is end of life. IPEX and IPEX-LLM are archived. OpenVINO is 2026.4.1 with a GGUF backend in llama.cpp (`GGML_OPENVINO`, device chosen by `GGML_OPENVINO_DEVICE`). vLLM has a built-in XPU platform. Xeon 6 reaches ⚠ 845 GB/s on twelve channels of MRDIMM-8800; AVX10.2 and APX arrive with Nova Lake and Diamond Rapids. **Apple.** M5 Max (128 GB, 614 GB/s) and M5 Ultra (512 GB, 1.2 TB/s) ship; M6 (32 GB, 170 GB/s) is in the Mac mini and 14-inch MacBook Pro. MLX is 0.32.4 with a CUDA backend on Linux and RDMA over Thunderbolt 5 for clustering. `vllm-metal` runs vLLM's scheduler on MLX. macOS 27 adds Core AI; it has no ONNX provider yet. **Google.** TPU 8t and 8i were announced 2026-04-22 (8i: 288 GB HBM, 8.6 TB/s); general availability is unconfirmed. Inference runs through vLLM's TPU platform (`tpu-inference`) and, in preview, TorchTPU. **Arm.** KleidiAI 1.33 carries SME2, SVE2 and SVE kernels and is in llama.cpp (`GGML_CPU_KLEIDIAI`), ONNX Runtime 1.31 and XNNPACK. Neoverse V3-class servers (Graviton5, Cobalt 200) have SVE2 and no SME2; Apple cores and Arm's C1/C2 mobile cores have SME2. The kernel now has a compute accelerator driver for Ethos-U. **Qualcomm.** llama.cpp has an OpenCL backend for Adreno and a Hexagon NPU backend (`GGML_HEXAGON`, experimental, v73 to v81, about 3.5 GB per NPU session). The QNN provider takes quantized, static-shape models. AI200 racks are sampling; AI250 is 2027. **Huawei.** Ascend 910B/910C are the installed base; 950PR/950DT are entering use; 960 is 2027. CANN is 9.2.1 and open source. `vllm-ascend` 0.30.0 tracks vLLM 0.30.0 with `torch_npu` 2.10; llama.cpp's CANN backend documents 910B and 310P. The ONNX Runtime CANN provider is community-maintained and covers convolutional models, not language models. **Others with a route.** Moore Threads (MUSA backend in llama.cpp, vLLM-MUSA, `torch_musa`), Cambricon, MetaX, Iluvatar and Enflame (vLLM plugins, each pinned to a different vLLM), AWS Trainium (`vllm-neuron`, Neuron SDK 2.33), Tenstorrent and FuriosaAI (their own vLLM-compatible servers), IBM Z (zDNN; Spyre through `vllm-spyre`). **The frameworks.** llama.cpp 1.0.0 (ggml 1.0.0), released 2026-11-13; ONNX Runtime 1.31.1, with DirectML in sustained engineering and execution providers moving to separately released plugins; the `ort` crate 2.0.0-rc.15; vLLM 0.33 with CUDA, ROCm, XPU, TPU and CPU platforms built in and a platform-plugin entry point for the rest; SGLang 0.5.23 with a device argument for `cuda`, `xpu`, `hpu`, `npu` and `musa`; PyTorch 2.15 with `torch.accelerator` as the vendor-neutral device API (memory information from 2.10). ## Quantum Surveyed 2026-11-12. ⚠ marks what came from secondary coverage. - **No quantum processor accelerates language-model inference.** The one demonstration on record (Multiverse Computing and IBM, arXiv:2605.05914, May 2026) put small quantum-circuit adapters into Llama 3.1 8B and ran inference end to end on a 156-qubit IBM Heron r2. It improved perplexity by 1.4% and its authors claim no quantum advantage; ⚠ one report puts a 129-token reply at about 90 minutes. A quantum processor is not an inference device, and this engine does not treat it as one. - **The traffic runs the other way.** Classical accelerators serve quantum hardware: CUDA-Q QEC 0.6 (April 2026) decodes error correction on GPUs in real time over NVQLink, and circuits are simulated on GPUs. That is work this engine's job engines are the right shape for. - **What the engine has.** Two job programs on NVIDIA CUDA-Q (0.16.0, 2026-09-12): `cudaq.ghz` samples a GHZ state and `cudaq.maxcut` runs QAOA on a graph. Each takes a simulator target. `qpp-cpu` is a CPU state-vector simulator and runs on any machine; `nvidia` and `tensornet` need an NVIDIA GPU, and CUDA-Q's GPU simulation is NVIDIA-only in practice. - **Vendor-neutral simulation exists, outside CUDA-Q.** Qiskit Aer has a ROCm build for AMD GPUs (⚠ `qiskit-aer-gpu-rocm`), PennyLane's `lightning.kokkos` builds with HIP for AMD, qsim has a HIP backend for Instinct, and QuEST builds with CUDA or HIP. None is confirmed on Intel or Apple GPUs. A simulation program on one of these is the way to a quantum job that does not need NVIDIA; none is written yet. - **Real devices.** Quantinuum Helios, IonQ Tempo and IBM's systems are reached through a provider's cloud, with credentials; some are also sold for installation on site (IQM reports 23 systems), and Google's Willow is open to selected researchers only. Whether any can be driven without a network by third-party software is not established. A job is confined without a network, so reaching a device is the same decision as the remote backend, for jobs: a separate, opted-in route that names its hosts. It is not built. ## Staying ahead What to watch, and the change each would call for. None is built on speculation; each is a note of where the next gap will open. | Signal | Why it matters here | Response when it lands | |---|---|---| | ONNX Runtime plugin execution providers | Providers stop being compile-time features of one runtime binary | Register provider libraries at run time; keep the planning rules | | Windows ML execution-provider catalogue | The system, not the application, supplies providers on Windows | Enumerate the catalogue instead of DirectML | | Core AI on macOS 27 | The Neural Engine's new front door; no ONNX path yet | A provider, when ONNX Runtime has one | | OCP FP8 E4M3 in ggml | The interchange type between vendors; the fork has it since the sync with 0.9.0 | Converters and loaders that carry it, once a model ships in it | | Block-scaled 4-bit formats diverging | MXFP4 (blocks of 32, E8M0 scale) and NVFP4 (blocks of 16, E4M3 scale) are not interchangeable; x86 ACE defines its own | Carry block size and scale format with a tensor, convert at load | | x86 AVX10.2, APX, ACE | Matrix and block-scaled arithmetic on CPUs from 2026 to 2029 | Detection in `praecise_hardware::cpu` once the standard library or CPUID leaf is settled | | A ratified RISC-V matrix extension | None is ratified; three proposals overlap | No path until one is | | Vendor KV tiers (CXL, high-bandwidth flash) | KV-cache placement becomes a tier choice, not a device choice | [foundations.md](foundations.md), consequence 3 | | A vLLM plugin converging on upstream | Fewer per-vendor engine environments | Collapse the environments that can share a vLLM | ## Changelog - **2026-11-13.** The llama.cpp fork is synced in source with upstream 1.0.0 (ggml 1.0.0). Upstream removed its deprecated API, so the bindings call the replacements it names (`llama_model_*`, `llama_vocab_*`, `llama_state_*`, `llama_init_from_model`, `ggml_prec_set_acc`), and `defrag_thold` is gone from the context parameters; the MoE cache size and its CPU share are now settable from Rust. Upstream dissolved `clip` into `mtmd`, and the fork's Inkling towers, Kimi-K3 projector and Qwen3-Omni talker generator moved with it. Built, on an x86-64 CPU with GCC: the merged tree (the library and `mtmd`), with upstream's architecture test passing 155 of 155 and its gradient test passing. Compiled: `llama-cpp-sys-2` and `llama-cpp-2` with the `mtmd` feature against it (a type-check, CPU). Not compiled: the other crates in the workspace, any GPU backend. Not run: the workspace's own tests. The CUDA attention and quantized-product kernels, which both upstream and the fork changed, were merged by reading. No speed figure is claimed for this release. - **2026-11-13.** The llama.cpp fork is synced with upstream 0.9.0 (ggml 0.29). Upstream's row-list attention and the fork's banded attention read the same tensor slot, and each reader now tells them apart by operation. Inkling and DiffusionGemma moved to upstream's attention-mask type. Built on the CPU (Windows x86-64, MinGW): the architecture test passes 152 of 152 and the gradient test passes. The crates that link llama.cpp type-check against it and the runtime's 272 library tests pass. Not compiled: any GPU backend, including the CUDA attention kernels the merge touched. Not run: the backend comparison tests, which need a second backend. Upstream publishes no speed figures for 0.8.0 or 0.9.0, and none was measured here. - **2026-11-13.** Versions refreshed: the lockfile and the four crates that had a newer release (`cc`, `encoding_rs`, `openh264`, `wgpu`); the cloud build image moved from Debian 12 to Debian 13; the CUDA test image moved to CUDA 13.5.0 on Ubuntu 26.04. Neither image has been built since the change. Every other declared crate was already at its newest stable release; `ort`, `dashmap`, `tokenizers` and `libc` have newer pre-releases only, and stay where they are. A November survey of decode speedups was added to [acceleration-research.md](acceleration-research.md): no new batch-1 kernel was found. - **2026-11-12.** Remote backend (`praecise-remote`) added as an opt-in route to API-only silicon. Quantum section added, with two open gaps: the remote backend against a real provider, and quantum simulation off NVIDIA. bindgen built without `prettyplease` so the workspace builds. - **2026-11-12.** Page written. Hardware layer, backend reach, OpenVINO hosted backend, vendor-neutral host adapter and confinement, six ONNX providers, five ggml features, CPU kernel tiers, the silicon catalogue, the bandwidth measurement and the `silicon-coverage` workflow added. Nine open gaps recorded. --- # Admission and queueing: why a 30-second deadline broke a 30 tok/s decoder **A serving layer that admits on predicted end-to-end time, against a fixed deadline, from a cost model that never saw the output length, will refuse almost everything on a slow decoder — and it will do so more, not less, the longer it runs.** Observed on a DGX Spark serving Qwen3.8-Flash-Next (UD-Q2_K_XL, 31.6 tok/s decode, 730 tok/s prefill) on 2026-09-02. Fixed by `praecise_runtime::admission`. ## What was observed A 200-token request, prompt "Reply with exactly: OK", was refused: ``` refused up front: predicted 48779 ms exceeds the 29999 ms deadline, so admitting it would miss and cost the requests already running reason: deadline-unattainable ``` Other clients on the same node saw the same, routed around it to a hosted model, and reported the local model as "burning its budget on hidden reasoning". It was not reasoning. It was not admitted. ## The three defects The host's admission was a reasonable design — admit on predicted deadline, order least-deadline-first, refuse what will miss rather than let it degrade everything else — fed by the wrong inputs. **1. Output tokens were not in the cost.** The estimate was `ms/kilotoken × prompt_tokens`, with prompt tokens approximated as bytes/4 and, on the OpenAI-compatible path, as a flat 1,000. On a decoder, cost is almost entirely output: a 20-token prompt generating 1,000 tokens is 32 s of work. Charged as 20 tokens, that one request taught the model "128 s per kilotoken". **2. The deadline was fixed while the work was not.** Thirty seconds end-to-end at 31 tok/s is ~900 output tokens. Every generation longer than that — every real coding turn — was predicted to miss and refused. A caller that asks for 4,096 tokens has asked for a two-minute answer; refusing because thirty is a round number protects no one. **3. Refusal where there should have been a queue.** The batching engine underneath holds a channel and admits into slots as they free. Turning callers away above it with a `retry_after_ms` they cannot act on discarded the queue that already existed. The only exemption was "admit if idle" — so the node served one request at a time and refused the rest. The comment in that code named the flaw itself: *"elapsed time measures sojourn while the prediction uses it as service time; until those are measured separately…"*. They are now. ## The model that replaces it ``` service_ms = prompt_tokens × prefill_rate + max_output_tokens × decode_rate × concurrency ``` - **Two rates, learned separately.** Prefill from time-to-first-token when the host reports it; decode from the remainder per generated token, normalised to a solo-equivalent by the concurrency it ran under (at c=2 on a bandwidth-bound decoder each request sees roughly half the solo rate, and the 52.5 vs 31.6 tok/s aggregate measured on this box bears that out). - **The SLO is on the wait, not the answer.** Interactive: 30 s in the queue before the first token. Batch: 600 s. Generation runs for as long as `max_tokens` allows. This is the convention of every production serving stack (TTFT and queue timeout; never a cap on total generation) and the only one under which long completions exist on a slow decoder. - **ETA before refusal.** With every slot busy, the answer is how long until one frees, computed from the remaining work of what is running (a request that overran its estimate counts as freeing now — the estimate was wrong, not the request). The host holds the caller and asks again as things finish. Only an ETA past the class budget, or a queue at its depth limit, is refused. A free slot always admits, whatever the estimate says. The caller chose the length; a free slot is capacity; there is nothing to protect. ## What it means for the numbers On the same hardware, same model, the 30-second interactive budget now buys a queue wait rather than a generation cap. What decode speed still governs is *how long the answer takes* — and that is where the rest of this repository's work (prefix reuse across agent turns, n-gram self-speculation) applies: at ~94 tok/s warm under concurrency 8 (sxuff's llama-server measurement for this model on one Spark) the same answer that took 32 s takes ~11 s. ## Tests `admission::tests` uses the measured rates above. The case that matters is `a_long_generation_does_not_poison_the_next_short_one`: after observing one 1,000-token answer, a 200-token request must still be predicted under 8 s. ## Choosing concurrency: a threshold, a validity gate, a pinned result **The transferable part of Nebius's Cosmos 3 Super study is its method, not its numbers.** Count only output that passes a validity gate, report it per hour of wall clock, choose concurrency by a marginal-gain threshold fixed in advance, and pin the exact configuration beside every figure. The topology result (eight single-GPU replicas beat one eight-GPU replica) is about splitting one diffusion forward pass across GPUs and has no counterpart on a single GB10. The method is implemented in `praecise_runtime::concurrency_sweep`, and is wired as an opt-in into the existing `praecise-runtime/tests/batch_throughput.rs` rather than a second tool. Source: Nebius, "Serving NVIDIA Cosmos 3 Super: Request Latency and Node Throughput", 2026-10-26, . Every figure quoted from it below was read from the post on 2026-10-29. None was reproduced here: no GPU benchmark was run for this note. ### What the study measured Cosmos 3 Super text-to-video (`nvidia/Cosmos3-Super`, one pinned revision) in `vllm/vllm-omni:cosmos3` (vLLM 0.25.0, pinned by digest), BF16, 1280x720, 189 frames at 24 fps, 35 denoising steps, guardrails off, one fixed prompt, three fixed seeds, 24 measured requests per topology, one warmup request per replica, on a single 8-GPU HGX B200 node and a single HGX H200 node. | Topology | B200 mean latency (s) | B200 valid video-s/node-h | H200 mean latency (s) | H200 valid video-s/node-h | |---|---|---|---|---| | 1 replica x 8 GPUs | 68.5 | 413.6 | 123.3 | 229.9 | | 2 x 4 | 121.5 | 466.3 | 230.0 | 245.6 | | 4 x 2 | 212.4 | 529.0 | 416.5 | 271.4 | | 8 x 1 | 378.1 | 589.4 | 779.0 | 289.1 | One 8-GPU replica to eight single-GPU replicas: +42.5% throughput on B200 and +25.7% on H200, at 5.5x and 6.3x the mean latency. A second request in flight per replica added 0.4 to 3.2% throughput (below the authors' 5% bar, set in advance) and 32.9 to 48.8% latency. All 408 recorded attempts passed the gate, so the gate changed no reported number. The post states its limits: one prompt, one shape, one software stack; the gate checks file integrity and basic motion, not prompt adherence or quality; B200 and H200 nodes differ in more than the GPU. ### What transfers **1. Rate-versus-latency is a curve with a knee, and the knee is found by measuring.** Both a video replica and a decode slot trade per-request latency for aggregate output. The shape of the trade differs (next section), but the procedure is the same: sweep, then stop where the next step stops paying. `concurrency_sweep::run_sweep` runs the levels in ascending order and records mean, p50 and p95 latency beside the rate at each. **2. A marginal-gain threshold, fixed before the run.** The study's rule was "under 5% throughput gain is not worth the latency". `choose` applies it as a stopping rule: take the highest level reached by consecutive steps that each gain at least the threshold (default 5%, `DEFAULT_GAIN_THRESHOLD`), and stop at the first step that does not, even if a noisier later level looks better. An optional latency budget caps the choice, which is the study's advice for intermediate latency budgets. The threshold is a parameter, not a finding: 5% is the authors' choice for their fleet and is the default here only for comparability. **3. A validity gate before output counts.** The study counted a clip only if it decoded fully, had the expected shape and was neither blank nor frozen; rejected attempts added no output but stayed in the wall clock. For an LLM the analogues are a request that completed, produced at least a minimum number of tokens, did not collapse to one repeated token, and did not decode to replacement characters. `LlmGate` implements those four; any other `Gate` can be passed. This is an integrity gate, not a quality gate: a fluent wrong answer passes, exactly as the study's gate does not judge realism. Where the engine already has a correctness check for a path (speculative decode acceptance, parity tests in `praecise-diffusion/tests/parity`), that check is the better gate for that path. **4. Valid output per hour, with the clock including failures.** `units / wall_s * 3600`, with `wall_s` from first dispatch to last finish. For a decode workload `units` is generated tokens. A concurrency level that wins on tokens per second but loses requests (timeouts, out-of-memory, a slot reset) is penalised automatically. **5. Pin the configuration with the number.** The study pinned the model revision, the container by digest, the engine version, the precision, the shape, the step count and the seeds. `RunPin` requires `model_digest`, `build`, `shape` and `hardware`, refuses to produce a report without them, and hashes the whole pin (`pin_digest`) so two reports can be compared by digest. For this engine the pin should carry at least: the GGUF SHA-256, the praecise-engine commit and the llama.cpp submodule commit, the CUDA build and the compiled architectures, the speculative method and block length, context size and slot count (`BATCH_MAX_SLOTS`), prompt and output lengths, seed or temperature, and the GPU memory other tenants held during the run. **6. Benchmark your own workload before committing.** The authors' last advice, and the repository's own history agrees: the 30-second deadline in [admission-and-queueing.md](admission-and-queueing.md) came from a cost model that never saw the real output length. ### What does not transfer **Replica topology.** The 1x8 versus 8x1 axis is tensor, CFG and Ulysses sequence parallelism against independent replicas. A GB10 is one device with one memory pool. There is no replica count to choose; the nearest axis is the number of engine instances sharing the pool, which the shared `Budget` in `praecise-host` already forbids duplicating for one model. **The shape of the concurrency curve.** In the study one request in flight per replica was the best operating point, because a diffusion replica runs one generation at a time and a second request only queues (the post says vLLM-Omni gave no real parallelism at two in flight). Slots in a batching decoder are not queued requests: they share one pass over the weights. Decode on GB10 is bandwidth-bound at about 80% of the 273 GB/s peak ([gb10-bandwidth-ceiling.md](gb10-bandwidth-ceiling.md)), so extra sequences in the same step add little read traffic and aggregate throughput does rise with slots. The repository already records two points of that curve for Qwen3.8-Flash-Next UD-Q2_K_XL on one Spark: 31.6 tok/s solo against 52.5 tok/s aggregate at two in flight ([admission-and-queueing.md](admission-and-queueing.md)), and about 94 tok/s warm at concurrency 8 (a llama-server measurement recorded in that note, not reproduced here). So the study's "one in flight is enough" does not carry over; what carries over is that the gain eventually falls under the threshold and the latency price keeps rising. Where it falls on GB10 for a given model is unmeasured here, and it depends on KV size per slot, context length and speculation (a drafted block per slot already raises the work per weight read). **Latency metrics.** The study uses time to a complete file because a synchronous video endpoint returns nothing before it. An LLM has time to first token and inter-token latency, and `Attempt.latency_s` is time to completion only. A sweep that feeds an interactive service should also record time to first token; that is a gap in this first version (see below). **Unified memory and other tenants.** The study had a whole node reserved. A GB10 shares one LPDDR pool, and bandwidth, with whatever else is running, so a sweep result is valid only for the load present while it ran. The pin must say what else held the device, and results measured next to a loaded neighbour should not be compared with results from an idle box. On the shared Spark any real run must go through the host's rules (capped build wrapper, the free memory floor, the exclusive GPU lock); `SweepBackend::admit` is the hook where an adapter enforces them, and a refusal stops the sweep at that level. **The 8-GPU parallelism vocabulary** (CFG parallelism, Ulysses-SP, HSDP) has no equivalent in the llama.cpp backend on one device. ### What exists in the repository already Inventory taken before building anything (2026-10-29, `main` at 9e72e77): - `praecise-runtime/tests/batch_throughput.rs`: an ignored test that runs the batch engine at several concurrencies and prints aggregate tok/s. It has no threshold, no validity gate, no latency, and asserts only that tokens were produced. It now has the opt-in sweep report described below; that path compiles but has not been run, because it needs a model and the shared device. - `praecise_runtime::admission`: a two-term cost model and queue-wait SLO. It decides admission at run time; it does not choose a slot count. - `BATCH_MAX_SLOTS` (default 32, `batching.rs`): the slot ceiling, reduced when whole contexts do not fit. Nothing chose the default by sweep. - `praecise_host::Engines` and `Budget`: one engine per model, one memory ledger. - `praecise-diffusion`: see the next section. Added, opt-in, without changing any default: `praecise-runtime/src/concurrency_sweep.rs` (13 CPU unit tests with a fake backend whose rate saturates at a chosen slot count) and a `PRAECISE_TEST_THROUGHPUT_SWEEP=1` mode in `batch_throughput.rs` that scores its existing levels through the gate and prints the report. ### Known gaps in this first version - No time-to-first-token or inter-token latency; completion latency only. - The `batch_throughput.rs` sweep mode is compiled but unrun (no GPU benchmark was run for this change); its latency is the engine's generation time, which excludes queue wait. On the shared Spark it must follow the host rules above and stay small. - Repeats and variance: one run per level, no confidence interval. The study reports repeat runs agreeing within 2.65% on its own harness; this version gives no such bound, so a gain close to the threshold should be repeated before it decides anything. - The gate is an integrity gate. Quality needs a separate evaluation. ### Tests `cargo test -p praecise-runtime --lib concurrency_sweep` (13 tests, no model, no GPU): the knee is found at the saturation slot count; a linear curve takes every step; a gain of 4.9% is refused and 5.1% taken; the walk does not resume after a failed step; the latency budget caps the choice; rejected output adds no rate but keeps its wall time; the rate formula; each gate check; an incomplete pin is refused; the pin digest is order independent and value sensitive; bad plans are refused before any run; a host refusal stops the sweep; the report round-trips through JSON. The curve in the tests is synthetic and says nothing about GB10. --- # Prefix reuse in the batch engine `BatchEngine` serves every running request from one context and a fixed pool of sequence slots. When a request finishes, its slot keeps what it computed, and the next request that shares a prefix with it starts where the two diverge instead of at position zero. A multi-turn chat or an agent loop repeats the whole conversation on every request, so this is the difference between prefilling a few hundred tokens and prefilling tens of thousands. The decisions live in `praecise_runtime::prefix_cache`, which is backend-free and tested without a model. `batching.rs` acts on them. ## What can be rewound is read from the model Whether a slot's cache can be cut back to a divergence is a property of the model's memory, read from llama.cpp at load (`llama_model_is_recurrent`, `llama_model_is_hybrid`, `llama_model_n_swa`, `llama_model_has_encoder`, `llama_model_has_decoder`, `llama_model_is_diffusion`, and the GGUF's `.attention.causal` and `.pooling_type`), never from the model's name. The result is a `MemoryTraits`, and `MemoryTraits::reuse()` a `Reuse`: | Memory | Examples | How a slot goes back to a shorter prefix | |---|---|---| | Attention (`Reuse::Trim`) | dense models; MoE models (experts do not change the cache); sliding-window models whose cache keeps every position | Trim every position past the divergence. | | Sliding window holding only the window (`Reuse::Window`, `BATCH_SWA_FULL=0`) | Gemma 3 | Trim while the window before the divergence is still held (checked against `llama_memory_seq_pos_min`); otherwise restore a checkpoint of the window. | | Recurrent state (`Reuse::Recurrent`) | Mamba, RWKV, gated delta nets, and every hybrid of recurrent layers with attention | Restore the deepest checkpoint at or below the divergence, then trim the attention part from there. | | Encoder-decoder (`Reuse::EncoderOutput`) | T5-style models | The context holds one encoder output; a request whose encoder input and cache namespace match it skips the encoder pass. Such a model decodes one request at a time. | | Bidirectional or pooled attention, and diffusion decoding (`Reuse::None`) | embedding and reranking GGUFs; masked-diffusion LLMs | Refused by `BatchEngine::spawn` with the reason. The engine generates text a token at a time from a causal cache, which these do not, and a prefix's states depend on what follows it, so nothing could carry over anyway. | `plan_rewind` turns (kind, cached length, shared prefix, checkpoints held, lowest live position) into `Keep`, `Trim`, `Restore` or `Reset`, and the engine verifies each rewind against the cache before it relies on it. ## Where checkpoints go A checkpoint is a copy of the part of a sequence's memory that cannot be trimmed: `llama_state_seq_get_data_ext` with `LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY`, which is the recurrent state or the sliding window. Checkpoints cost memory, so they go where a following request actually diverges: at the start of a chat turn. At load the engine renders a short probe conversation through the model's own chat template and reads the control token that closes each turn (plus every end-of-generation token); a turn starts right after one (`TurnEnds`). While prefilling, a slot's prefill stops at each chosen turn start, so the snapshot is taken exactly there. `checkpoint_cuts` always keeps the last turn start (where the reply being generated begins, and where the next request re-renders that reply, with its reasoning dropped or a stop sequence trimmed) and one token before the prompt's end (an identical re-send then prefills a single token), then earlier turn starts, latest first, at least 256 tokens apart. ## How many checkpoints memory affords Checkpoints are host memory. At load the engine measures one checkpoint and one token of KV on the live context, reads the host's free memory from the backend device list, and gives checkpoints half of what is free past a reserve (a tenth of total, at least 2 GiB), split across slots (`checkpoint_budget`), at most 32 per slot or `BATCH_PROMPT_CHECKPOINTS` (0 turns them off). Where the KV cache and checkpoints draw on the same memory (CPU serving, integrated GPUs, Apple silicon, and superchips whose GPU reports the host's memory as its own), a context too large to leave every slot its two most useful checkpoints is recreated smaller until it does, never below a quarter of what was asked or 8K tokens (`context_to_release`). On a discrete GPU the KV cache lives in device memory and the two do not compete. Both readings come from the backend's device list at load, so nothing is keyed to a particular machine. An operator can override them: `BATCH_CHECKPOINT_MEMORY=` fixes the memory checkpoints may use across every slot, and `BATCH_UNIFIED_MEMORY=1` (or `0`) says whether that memory is shared with the KV cache. ## Reasoning and stop sequences A slot's identity is the prompt as it was prefilled plus the tokens it generated. The next request usually re-renders that reply differently: the reasoning dropped by the template, a stop sequence trimmed, or the text retokenized. The divergence then falls inside the last turn, and the engine rewinds to it: attention models trim there, recurrent models restore the last-turn-start checkpoint. `ChatMessage::reasoning_content` lets a template that renders reasoning into history reproduce the previous turn exactly, so it stays an exact prefix. ## Media An image or audio attachment is part of the prompt's identity by content: the SHA-256 of its bytes is its bitmap id, and each of its positions gets an identity element derived from that hash (`media_ids`). A later request reuses through an attachment only when it carries the same bytes at the same place, and a reuse never ends inside one (`snap_out_of_media`). Multi-axis RoPE gives a chunk a different number of KV positions than identity positions; `kv_pos` translates between the two. Media chunks are evaluated one at a time (`MtmdInputChunks::eval_chunk`), so chunks a slot already holds are never re-encoded, and the text around them is prefilled in the ordinary batch. ## Cache namespaces `GenerationConfig::cache_salt` is an opaque isolation key chosen by the caller. A slot's cache is tagged with the namespace it was filled under (`Namespace`), and a request only matches slots of its own namespace. When nothing is reachable, it takes an empty slot before any other namespace's cache, then the least recently used one, so whether a prompt under another key is resident changes neither what is reused nor where the request lands, and cannot be read from time to first token. Unset, requests share one namespace. ## What a result reports `InferenceResult::cached_tokens` is the number of prompt tokens the request took from the cache instead of prefilling. A request with `commitment_k` set gets its top-k logit commitment from the batch engine, one record per generated token, speculative rows included. The minimum `k` is 4 (`toploc::MIN_COMMITMENT_K`); a smaller request is raised to it, and a verifier refuses a commitment whose `k` is outside 4 to 64, whose steps carry other than `k` logits, or whose emitted token is not in the recomputed top-k at its step. A prompt that ends in a media chunk takes its first logits from a row the engine cannot read, so such a request returns no commitment rather than a partial one. ## Streaming never waits on a reader Each sequence streams through `StopStream::nonblocking()`. A chunk the receiver's channel cannot take yet is queued for that sequence alone and retried every step. A receiver more than `STREAM_BACKLOG_LIMIT` (1 MiB) behind ends its own request with an error, and a finished request's result is delivered once its stream has been fully taken, or after 60 s. One slow reader never holds back the scheduler thread. A request may also set `GenerationConfig::reasoning_tx`: the model's reasoning is then delivered there as it is produced, kept apart from the answer on `token_tx`, instead of only in `InferenceResult::thinking` at the end. It is held and retried the same way, and a reasoning receiver that goes away ends only the reasoning stream. The speculative and layer-pipeline decode paths honour the same field. ## A bounded request queue `BatchEngine::submit` never blocks and never queues without bound. Behind the running sequences at most one slot's worth of requests may wait (each holds its whole prompt and any media bytes); past that, `submit` returns `Error::QueueFull` and the caller's admission decides whether to hold or shed. ## What is billed `InferenceResult::output_tokens` counts the tokens the model generated. When a reasoning budget runs out the engine forces the block's close marker into the output; those tokens appear in the text and in the commitment's steps, but are not counted as generated. ## Measured One small GGUF per memory kind, served on CPU with 4 slots and a 4096-token context, greedy decoding, 12 output tokens. Every request was also sent to a second engine that had never seen it, and the two answers were compared token by token through their top-8 logit commitments; a difference is accepted only as a near tie (the other engine's token within 1.0 logit of the top), which is the spread two runs of an identical prompt show with nothing reused. `cached` is the prompt tokens not prefilled, out of the prompt's length. | Memory kind (model) | identical re-send | exact extension | divergence at a turn boundary | same prompt, other namespace | prompt tokens saved | |---|---|---|---|---|---| | dense (Qwen3 0.6B) | 171 / 172 | 183 / 188 | 168 / 192 | 0 / 172 | 690 of 1208 | | sliding window (Gemma 3 270M) | 164 / 165 | 176 / 182 | 165 / 185 | 0 / 165 | 670 of 1161 | | hybrid gated delta net (Qwen3.5 0.8B) | 170 / 171 | 182 / 187 | 163 / 191 | 0 / 171 | 678 of 1201 | | Mamba (130M) | 163 / 164 | 175 / 182 | 163 / 181 | 0 / 164 | 664 of 1153 | | RWKV-7 (0.1B) | 163 / 164 | 175 / 180 | 163 / 182 | 0 / 164 | 664 of 1150 | | MoE (Granite 3.1 1B-A400M) | 168 / 169 | 180 / 187 | 169 / 189 | 0 / 169 | 686 of 1190 | | vision (SmolVLM 256M) | 303 / 304 | 305 / 311 | 304 / 323 | 0 / 304 | 1222 of 2280 | | hybrid vision (Qwen3.5 0.8B + projector) | 183 / 184 | 185 / 190 | 176 / 204 | 0 / 184 | 720 of 1443 | | encoder-decoder (Flan-T5 small) | 185 / 185 | 0 / 191 | 0 / 200 | 0 / 185 | 185 of 1251 | | embedding (EmbeddingGemma 300M) | refused at spawn | | | | n/a | On the encoder-decoder model what carries over is the encoder output itself: an identical input skips the encoder pass entirely (so it reuses the whole prompt, not all but one token), and any other input, an extension included, reuses nothing, because a bidirectional encoder's output for a prefix depends on what follows it. Another namespace sending the same input re-encodes. The same image replaced by a different one reused exactly the text before it on the attention model (155 of 304) and nothing on the recurrent one, which keeps no checkpoint below the image. Two conversations served together answered as each did alone on every model. A build whose trim reports success without removing the stale positions fails the same test on its first divergent request. --- # Request shaping: what a layer above the backend can still control Decode on GB10 is bandwidth-bound at ~80% of peak (see [gb10-bandwidth-ceiling.md](gb10-bandwidth-ceiling.md)), so neither speculative allocation nor host-side scheduling has much room. What remains reachable from *outside* a backend is how requests are **shaped and grouped** before they reach it — and a few of those levers are surprisingly sharp. Every item here is arithmetic about batching, not a theory about text. That is why they are worth more than the allocation work: nothing has to be true about the model for them to hold. ## 1. One request can make a whole batch expensive `all_greedy` and `no_penalties` are **batch-wide booleans** in vLLM. A single request using frequency/presence/repetition penalties drags the *entire batch* onto the expensive sampling path, including every request that asked for none. This is the highest-leverage item here and it is invisible from inside a single request. A layer that groups penalty-using requests together — or that drops penalties a caller set to a no-op value — keeps the cheap path cheap for everyone else. ⚠ Verified in vLLM source, and cited in no secondary write-up found. Do not expect a backend to do it for you. ## 2. Penalties dominate host cost; the sampler does not Measured at batch 4: **6-47 ms/step for penalties**, growing with context length, against **~43 us for the sampler itself** — three orders of magnitude apart. The intuition that a 150k-250k vocabulary makes sampling expensive is wrong. It is the penalty bookkeeping over the generated sequence that costs, and it grows as the sequence does. ## 3. Prefer `min_p` to `top_p` `min_p` is softmax + amax + compare, with **no vocabulary sort**. `top_p` requires the sort. Where a caller's intent is "drop implausible tokens" rather than "keep a fixed probability mass", `min_p` expresses it and costs less. ## 4. Round chunk sizes to a multiple of 256 Tile quantisation: a chunk size of **257 costs 32% more than 256**. Trivially enforceable from outside, and a caller will almost never notice the difference between the two values they asked for. ## 5. Prefix reuse is worth structuring prompts for Production hit rates of **52.4% and 74.1%**, worth **1.7x on TTFT** — but capped near ~50% on long-context traffic, so treat 50% as the planning figure rather than the headline. Mechanics that decide whether it works: - Block granularity is **16 tokens**, so a shared prefix must align to that boundary to count. - **Sampling parameters do not break the cache hash. LoRA does.** - `cache_salt` is a per-request lever for deliberately *avoiding* reuse. Structure prompts with the stable part first — system prompt, tools, few-shot exemplars — and the variable part last. That is a prompt-construction decision, which is squarely a layer-above concern. ⚠ A "91% production hit rate" for Mooncake circulates and is **refuted by the paper's own Table 1**. Use ~50%. ## What does NOT apply here, and why **Most host-side optimisation is measured on host-bound H100 serving.** GB10 decode at 22.7-38.7 tok/s gives a **~44 ms step budget**, against which detokenisation is under 0.01%. At batch 1 on this machine, host work is not the constraint — the memory bus is. Before adopting anything from the host-overhead literature, measure whether the workload is actually CPU-launch-bound. It very likely is not, here. ⚠ Two cautions on citing this area at all: - The widely-repeated "GPU is only 38% of a decode step; scheduling 29%, API server 33%" breakdown **has no locatable source**. The ~50% CPU-overhead claim is real and traces to the SGLang v0.4 blog; that specific triple does not. It was cited several times during this work before being checked. - One vLLM host-side optimisation **decayed from a real 5.1% win to 0.20% over ~1,640 commits** and was rejected. Re-measure any published delta against current main rather than trusting the number in its PR. - Async scheduling **cannot be combined with structured outputs**. ## An untested cheap experiment llama.cpp has **no big.LITTLE awareness on ARM**. The GB10 has 10 Cortex-X925 performance cores and 10 Cortex-A725 efficiency cores; threads are placed without regard to which. Pinning to the X925 cores only is a cheap experiment, and **no published ARM/Grace threading or core-pinning study for LLM decode exists** — so it is genuinely unmeasured rather than known-useless. --- # Adding a diffusion family to Praecise Engine **Status:** the first family is implemented as the `praecise-diffusion` crate: FLUX.2 [klein] (rectified-flow transformer, Qwen3 prompt encoder, KL autoencoder), text-to-image and reference-image conditioning, bf16 or load-time 8-bit weights, on the same ggml build, device and backend as the language-model family. On a host with GPU hardware it refuses to run on the CPU. It runs in the process: an application claims the model in `praecise_host::Engines` with `Flux2Klein::resident_bytes()` before loading it, so one model is served by exactly one engine within one GPU memory budget. `examples/flux2` generates and reports per-stage timings; `praecise-diffusion/tests/parity` checks every stage against the reference implementation on a small random checkpoint. The design notes below were written after serving LTX-2.5 (22B video DiT) alongside Praecise Engine-accelerated LLMs on a GB10. > **Naming.** *Praecise* is the family. **Praecise Engine** is the acceleration > layer — it makes a model faster. **praecise-harness** is the agentic > framework — it makes an agent work. They sit at different levels and are used > independently. This document is about Praecise Engine. ## The problem Praecise Engine is backend-agnostic acceleration on top of an inference backend, llama.cpp first (linked), with vLLM, SGLang, TensorRT-LLM and transformers reached as hosted backends. That is agnosticism across **backends**. Every backend on that roadmap is an **autoregressive token generator**. A diffusion model is not a different backend for the same task. It is a different task, and the public surface says so: ```rust pub struct GenerationConfig { pub temperature: f64, pub top_p: f64, pub max_tokens: u32, pub repeat_penalty: f32, pub repeat_last_n: usize, pub seed: u64, pub top_k: Option, pub min_p: Option, pub frequency_penalty: f32, pub presence_penalty: f32, pub stop: Vec, pub draft_n: Option, pub commitment_k: Option, } pub struct InferenceResult { pub text: String, pub thinking: Option, pub input_tokens: u32, pub output_tokens: u32, pub tokens_per_second: f64, pub stop_reason: StopReason, pub commitment: Option, } ``` One field — `seed` — means anything to a denoiser. There is no token to sample, no distribution to shape, no stop string, no KV cache, no draft model to verify against. `StopReason` has no referent: a diffusion run ends because the schedule ended. The question is whether Praecise Engine takes a **second task family**. That is a larger commitment than a second backend. ## What is shared 1. **Residency and admission.** The binding constraint is not decode speed, it is which weights may occupy memory. On a 121.7 GiB unified-memory GB10 a 22B video DiT needs a 66.5 GiB serving set against an LLM holding 41.6 GiB under an 88 GiB ceiling. That arithmetic is identical in kind for both families, and today neither Praecise Engine nor its consumer owns it — it leaks into node config. 2. **The hardware-tuning harness.** The machinery for measuring and selecting per device. Not the tunings themselves. 3. **Load-time numeric casts.** `fp8-cast`-style downcasting is the same plumbing for a decode loop or a denoiser. 4. **Determinism.** Seeded, reproducible runs matter more for media than text. 5. **Verifiable inference** (`toploc`), if it stays — see open questions. ## What is not shared Sampling, speculative decode, KV cache, sequence batching, stop-string streaming. `batching.rs` batches *sequences*; diffusion batches *latents* across a fixed step count — a different scheduling problem with a different cost model. ## Acceleration conclusions invert between the families This is the finding that shapes the design. | technique | autoregressive decode | video diffusion | |---|---|---| | weight-only NVFP4 | wins — bandwidth-bound | loses — ~33% smaller, ~6% *slower* | | speculative decode (DFlash/MTP) | 1.85–2.13x measured | n/a — nothing to draft | | step caching (TeaCache) | n/a | wins | | attention sparsification (STA) | limited | wins — ~3x reported | LLM decode is memory-bandwidth-bound, so shrinking weights buys time. Video diffusion is compute-bound, so shrinking weights buys an unpack penalty. NVIDIA's published "20% faster, 40% memory savings" for LTX-2.5 is NVFP4 *plus* FastVideo kernels, STA and TeaCache. The kernels are the speedup, not the format. **The shared layer is therefore the harness — measure, select, record — never a table of techniques presumed portable.** A `praecise-core` exporting "use NVFP4 on Blackwell" is wrong half the time. ## Shape ``` praecise-core error, seed/determinism, device + memory budget, tuning harness, telemetry, (toploc?) praecise-runtime autoregressive LM family — today's crate, API unchanged praecise-diffusion iterative-denoiser family — new ``` Additive. `praecise-runtime` keeps its public API verbatim. The families share `praecise-core` and nothing else, each owning its config and result types: ```rust pub struct DenoiseConfig { pub steps: u32, pub guidance_scale: f32, pub schedule: SigmaSchedule, // distilled checkpoints ship fixed ones pub seed: u64, pub cast: Option, // fp8-cast et al, load-time } pub struct DenoiseResult { pub steps_run: u32, pub time_ms: u64, pub steps_per_second: f64, } ``` No `StopReason`, no token counts, no sampler. Unifying these with `GenerationConfig` behind one trait produces a struct where most fields are `None` for any given caller. ## What this has to earn A diffusion path already exists outside Praecise Engine: a Python/diffusers worker with a `FamilyAdapter` seam, a job marketplace, and per-family conversion fixes. A second implementation is justified by one of: - **In-process serving** — no Python worker, no IPC, for a node already linking Praecise Engine. - **One memory budget across families**, so a DiT and an LLM contend through the same admission control rather than two schedulers that cannot see each other. This is the strongest case, and the exact failure observed: 66.5 + 41.6 GiB against an 88 GiB ceiling, neither side aware of the other. - **Verifiable media inference**, if commitments extend to latents. ## Staging 1. Extract `praecise-core` from `praecise-runtime` with no behaviour change — error, seed, device/memory budget, tuning harness. `praecise-runtime` depends on it and keeps its API. 2. Land the memory/admission model in `praecise-core`; the LM family uses it. This carries its own value independent of a second family. 3. `praecise-diffusion` behind a feature flag, one model (LTX-2.5 dev bf16), no kernel work — correctness first, `fp8-cast` and CPU offload as the levers. 4. Kernels (NVFP4-aware linears, STA, TeaCache) last, against measured baselines, given the inversion above. Step 3 is the commitment point, and it is reversible. ## Open questions - **`toploc` placement.** `InferenceResult.commitment` and `toploc.rs` are the largest non-batching module, and verifiable-inference is slated to leave Praecise Engine as a consumer concern. Settle this before a second family inherits the ambiguity. - **Does `batching` generalise?** Latent batching across a fixed step count may reuse the slot accounting. - **ARM.** FastVideo ships no prebuilt aarch64 wheel; kernel work on GB10 builds from source. ## Serving a video world model: what is hosted today and what a service would need Inventory (2026-10-29, checked against the code). The Nebius study discussed in [admission-and-queueing.md](admission-and-queueing.md) served a video model as a service; this repository hosts video models in two narrower ways: - **In process:** `praecise-diffusion` contains a `Cosmos3` implementation (`src/cosmos3.rs`, with `src/cosmos3/`), exposed through `examples/cosmos3`, which takes a checkpoint directory, a prompt, optional first frame, frame count and run count and reports per-stage timings. `wan-world`, `ltx2` and a `matrix_game` module sit beside it. These are examples and library code: nothing here is a long-running, multi-request video server, and the examples run one generation at a time. - **As a job:** `praecise_host::job` runs `cosmos.generate` (Cosmos 3 Edge or Nano through the Python `diffusers` pipeline in a pinned environment, with the Cosmos guardrail forced on). One run per process, confined, to completion. - **Not present:** the Cosmos 3 Super model the study used, a request queue for video, multi-GPU parallelism, or any measurement of any of it against this study. No claim is made here that the engine reproduces the study's throughput or latency. If the engine were to serve a video world model as a service, the study says what the serving path must have: 1. **One in-flight generation per resident copy** unless a measurement shows otherwise, with the queue outside the model; the second request should wait, not share. The `Engines` claim already guarantees one copy. 2. **A memory ledger that sees the whole serving set.** `diffusion-family.md` records a 66.5 GiB video DiT serving set beside a 41.6 GiB LLM (108.1 GiB against an 88 GiB ceiling on one GB10, neither side aware of the other); the `Budget` reservation must include text encoder, VAE and activations, not weights alone. 3. **A video gate:** the output decodes fully, has the expected frame count, width and height, and is neither blank nor frozen. `Gate` takes any checker; the video one would use `praecise-codec`. It does not exist yet. 4. **Latency as time to a complete file, and a throughput unit of valid video-seconds per hour**, which is `units` in `Attempt` with `units = seconds of valid video`. 5. **Throughput accounted per whole run:** warmup, model load and encode in or out of the timed boundary stated in the pin, as the study did (it excluded startup and load, included routing, generation and MP4 encoding). 6. **Guardrail state in the pin.** The study ran with guardrails disabled; the `cosmos.generate` job forces them on. A comparison across the two is not like for like. 7. **A topology decision only on hardware with more than one device.** On a single GB10 the question does not arise. --- # Foundations: what sits under the silicon Chip vendors are the visible layer. Under them, a small number of foundries, packaging lines, memory makers and interconnect standards decide what silicon can exist, how much memory it carries and how fast that memory is read. An engine that plans only against today's vendor lineup is planning against the output of those decisions and not the decisions. This note records the state of that layer in November 2026 and, for each part, the consequence for this engine. The premise it is read against is the one this engine is built on: **decode is memory-bandwidth-bound** ([gb10-bandwidth-ceiling.md](gb10-bandwidth-ceiling.md)), so memory and the links between memories are the first-order facts. ⚠ Most of this comes from trade press and analyst estimates gathered on 2026-11-12, not from primary documents. Roadmap dates slip. Treat every figure as a ranking aid and every date as provisional; where a standard body or vendor published the fact, the source is named. ## Process - **TSMC** makes nearly every leading accelerator. N3 is the binding wafer constraint for 2026 AI silicon; N2 has been in volume since late 2025; A16 (backside power) slipped to 2027; A14 is 2028. - **Samsung** ships SF2 in Exynos 2600 and has Tesla's AI6 on SF2P for late 2027. **Intel** ships 18A in Panther Lake; 14A is 2028 to 2029. - **SMIC** reaches a 5 nm-class node (N+3, below TSMC N5 density) by DUV multipatterning, without EUV. Huawei's Ascend line depends on it. **Consequence.** Process sets compute per watt; it does not move the decode ceiling, which memory sets. Two fleets follow from it: one on leading nodes with HBM4, one in China on 7 nm and 5 nm-class nodes with scarcer, slower memory. The second is not a slower copy of the first. It needs the INT8 and FP8 paths, aggressive weight compression and CPU fallbacks kept as well tested as the FP4 paths. ## Packaging - **CoWoS** capacity, not wafers, limits how many accelerators are built, and the interposer limits how much memory each carries: about 12 HBM stacks at the current size, with larger interposers targeted for 2027. Warpage reportedly forced one 2027 design back from 16 stacks. - Panel-level packaging and glass substrates are 2027 to 2029. **Consequence.** Memory per accelerator plateaus (288 GB, 432 GB, then about 768 GB to 1 TB) while memory per rack grows. A model that does not fit one device does not wait for a bigger device; it spans devices. Sharding, and emitting more tokens per pass over the weights, stay the main levers. ## Memory This is the part that bounds the engine directly. | Tier | Capacity | Bandwidth | State | |---|---|---|---| | HBM4 (JEDEC JESD270-4) | 36 GB per 12-high stack; to 64 GB at 16-high | 2 TB/s per stack at the base pin speed | shipping | | LPDDR5X SOCAMM2 | 192 GB modules; 1.5 to 2 TB per server CPU | about 1.2 TB/s per CPU | shipping | | DDR5 MRDIMM | per channel | about 70 GB/s (8800), about 102 GB/s (12800) | 8800 shipping; 12800 near standard | | LPDDR6 (JEDEC JESD209-6) | to 512 GB per module in the next revision | 10,667 to 14,400 MT/s | announced | | CXL 4.0 | expansion, not pooling, in real deployments | bounded by the link | expansion shipping | | High-bandwidth flash (OCP) | to 512 GB per stack | 0.4 to 3 TB/s by grade, read-mostly | announced | - All three HBM makers are qualified for the current generation; HBM4E with customer-specific base dies is 2027. - DRAM and NAND are short through 2027, with prices rising each quarter. Client devices are being built with less memory than 2025 roadmaps assumed. - Vendors are building a tier for KV cache between memory and storage. **Consequences.** 1. **Model memory as tiers, not as "VRAM and RAM".** Weights, live KV cache and cold KV state have different read rates and belong in different tiers. `praecise_hardware::silicon::MEMORY_TIERS` is the start of that table. 2. **Measure bandwidth; do not infer it from a part's name.** Pin speed varies by vendor and bin, MRDIMM against RDIMM changes a socket by half again, and CXL memory sits behind a slower link. `praecise-hw measure` measures what a machine's CPU threads reach. 3. **KV-cache placement is a policy of its own.** Prefix checkpoints already live in host memory ([prefix-reuse.md](prefix-reuse.md)); the same interface should take a slower, larger tier when one exists, with the cost of restoring set against the cost of recomputing. 4. **Plan for less memory on client devices.** Sub-4-bit weights, partial residency of mapped weights and small default contexts matter more, not less. ## Interconnect - **Inside a rack**: NVLink 6 at 3.6 TB/s per GPU; UALink 2.0 published April 2026, with 1.0 silicon in products in 2027; AMD's Helios ships now on UALink over Ethernet. Ultra Ethernet 1.0 is published and shipping in NICs. - **Between cards**: PCIe 6 is the shipping generation; PCIe 7 is specified; PCIe 8 is a draft for 2028. - **Between desktops**: RDMA over Thunderbolt 5 on macOS 26.2 and later, in a full mesh, at about 50 to 60 Gb/s. Apple reports tensor-parallel decode of an 816B model across four machines on it. **Consequence.** The transport is not one thing and will not converge before 2027. A multi-device layout should be given a topology (hops, bandwidth, latency) and not assume a fabric. The engine's layer pipeline (`praecise_runtime::pipeline`) splits whole blocks across machines on a LAN; which split pays on which link is a measurement to take per topology, not a rule to derive here. A rule drafted for this note, that a slow link forces a whole-layer split, was dropped because published Thunderbolt results contradict it. ## Datatypes and instruction sets - **OCP Microscaling (MX)**: MXFP8, MXFP6, MXFP4 and MXINT8, blocks of 32 with a shared 8-bit scale. **NVFP4** uses blocks of 16 with an E4M3 scale. The two 4-bit formats are not interchangeable. AMD CDNA 4 and NVIDIA Blackwell implement block-scaled formats natively. - **IEEE P3109** (arithmetic formats for machine learning) is an unapproved draft with no silicon. - **x86**: ACE (matrix multiply into AMX-style tiles, block-scaled FP4) and AVX10.2 are specified; no CPU with either has shipped. First parts are expected from late 2026 (AVX10.2, APX) to 2028 or later (ACE). - **Arm**: SME2 is on Apple M4 and M5, Exynos 2600 and Arm's C1-Ultra; server cores of the Neoverse V3 class have SVE2 without it. - **RISC-V**: RVA23 silicon ships; no matrix extension is ratified. **Consequences.** 5. **Block-scaled 4-bit is a family, not a type.** A tensor has to carry its block size and scale encoding so that one artefact can be converted at load for the device it lands on. 6. **FP8 E4M3 is the interchange type** between vendors; the fork has it since its sync with llama.cpp 0.9.0. 7. **CPU detection covers what ships.** `praecise_hardware::cpu` reports AVX-512, AVX-VNNI and AMX, the Arm dot-product, int8-matmul, SVE and SME families, RISC-V vector and LoongArch LSX/LASX. AVX10.2, APX and ACE are added when silicon exists to test against. ## Power Racks are power-capped (about 120 to 140 kW for current flagship racks, with 600 kW designs announced), and tokens per watt is becoming the figure buyers compare. MLPerf Power validates wall power for a whole system; there is no standard tokens-per-joule metric. **Consequence.** Throughput reported without energy is half a measurement on hardware that is bought by the megawatt. Energy per token belongs beside tokens per second where a machine exposes power, labelled with where the reading came from. ## Compute that sidesteps the premise Wafer-scale and SRAM-centric parts (Cerebras, Groq, d-Matrix) hold weights in on-chip memory, so their bound is capacity and compute, not DRAM bandwidth. They are reached over an API today, which this engine does only through its opt-in remote backend ([silicon-coverage.md](silicon-coverage.md)), where the provider's hardware cannot be measured or steered. If one becomes locally addressable it is a different class of device: small, latency-critical models, such as a drafter for speculation, are what it is for. ## Sources Standards: JEDEC (JESD270-4 HBM4, JESD209-6 LPDDR6), the CXL Consortium (CXL 4.0), the UALink Consortium (UALink 2.0), the Ultra Ethernet Consortium, the Open Compute Project (Microscaling Formats; the high-bandwidth flash workstream), PCI-SIG. Vendor announcements: TSMC, Samsung, Intel, AMD, Apple, SK hynix, Micron, Sandisk. Analysis and reporting: TrendForce, SemiAnalysis, Tom's Hardware, The Next Platform, ServeTheHome, Phoronix. --- # Measured draft acceptance, and what it refutes **The first non-simulated measurement in this work.** Run 28 Aug 2026 on an RTX 5070 Ti (16 GB, discrete), Qwen3-0.6B Q8_0, llama.cpp b10667, n-gram self-speculation (`--spec-type ngram-mod`), greedy decoding, 8 prompts spanning structured text through open prose. ## The numbers ``` draft tokens: 598 accepted: 556 overall acceptance 93.0% draft events: 12 ``` Per-position acceptance, out of 12 draft events: | position | accepted | rate | |---:|---:|---:| | 0-4 | 12 | **100%** | | 5-20 | 11 | **92%** | | 21-28 | 10 | **83%** | | 29 | 9 | **75%** | ## What it refutes `prax::affordable_block` prices a draft block as `a^k` -- geometric decay, on the reasoning that position `k` is reached only if every earlier position matched. That reasoning is sound and the model is still wrong, because acceptance at each position is **not independent**. Once a drafter has locked onto a repetitive or structured passage it keeps being right. | position | measured | `a^k` predicts | error | |---:|---:|---:|---:| | 0 | 1.000 | 0.930 | +0.070 | | 5 | 0.917 | 0.646 | +0.271 | | 10 | 0.917 | 0.449 | +0.468 | | 20 | 0.917 | 0.217 | +0.700 | | 29 | 0.750 | 0.113 | **+0.637** | **At position 29 the geometric model is off by 7x.** It predicts 11% where the drafter actually delivers 75%. The practical consequence runs one way: the closed form makes PRAX draft **far shorter blocks than it should**, so it leaves throughput on the table rather than wasting verification. Every simulated result in [praecise-method.md](praecise-method.md) rests on this model, and is therefore pessimistic about long blocks. Two incidental findings: - llama.cpp drafted **past `--spec-draft-n-max 8`** -- the histogram runs to position 29 -- so that flag is not the hard cap it reads as. - **12 draft events produced 556 accepted tokens.** On structured text an n-gram drafter is extraordinarily effective, which is the case for `crate::ngram` needing no model at all. ## Second run: Qwen3.5-0.8B, the model actually in use Repeated on `qwen3.5-0.8b` -- already present on the host, and the model chosen over 0.6B -- with the same prompts, drafter and settings. ``` draft tokens: 397 accepted: 397 overall acceptance 100.0% draft events: 8 ``` **Every drafted token was accepted.** And yet the per-position curve still decays: | position | accepted of 8 | rate | |---:|---:|---:| | 0-9 | 8 | **100%** | | 10-18 | 7 | **87.5%** | | 42 | 6 | **75%** | | 43-63 | 5 | **62.5%** | Note it runs to **position 63** -- llama.cpp drafted eight times the nominal `--spec-draft-n-max 8`. ### This kills the geometric model from both directions | position | measured | `a^k` at a=1.00 | `a^k` at a=0.93 | |---:|---:|---:|---:| | 9 | 1.000 | 1.000 | 0.484 | | 18 | 0.875 | 1.000 | 0.252 | | 43 | 0.625 | 1.000 | 0.041 | | 63 | 0.625 | 1.000 | 0.010 | Fit `a` to the *overall* acceptance of 100% and the model predicts no decay ever -- it would draft without limit. Fit it to the observed tail and it collapses immediately, drafting far too short. **No single `a` reproduces the shape**, because the shape is not geometric. ### What the shape actually is A **step function**: long flat runs punctuated by sudden drops. That is exactly what an n-gram drafter does -- it holds a pattern until the pattern ends, then loses several positions at once. Independence is the wrong model not because the decay rate is wrong but because the decay is not smooth. This is the strongest argument for [`AcceptanceProfile`](../praecise-runtime/src/prax.rs): a measured per-position curve represents a step function natively, where no closed form with one parameter can. `DECAY_DAMPING` remains only as the prior used before enough positions have been observed. ## What this does NOT establish - **One model, one drafter, one machine.** Qwen3-0.6B with n-gram self-speculation on a 5070 Ti. A larger target, a trained drafter (EAGLE, DFlash) or a different task mix could all look different. - **12 draft events is a small sample.** The shape is clear; the exact rates are not tightly bounded. - **Prompts skewed structured.** Half were code, JSON, SQL or literal repetition, chosen deliberately because that is where an n-gram drafter earns its place -- but it flatters the acceptance figure relative to open prose. - **No tokens/sec comparison.** This measures acceptance, not speedup. Whether a better block-size model produces a faster server is still unmeasured. ## Reproducing it llama.cpp b10667 (CUDA 12.4 build) serving the model with `--metrics` and `--spec-type ngram-mod`, greedy decoding, `n_predict 200`. Bind the server to a port nothing else is using, and put the working directory on a volume with room -- both sound obvious and both cost a retry here. Read the curve from `/metrics`: `llamacpp:spec_decode_num_accepted_tokens_per_pos_total`. ⚠ Fetch it with `System.Net.WebClient`, not `Invoke-WebRequest` -- the latter throws `NullReferenceException` trying to parse Prometheus text as a DOM. ⚠ GitHub's `/releases/latest` for llama.cpp resolves to a **nightly tag holding only a text file**. Walk `/releases` and match `^llama-.*bin-win-cuda`; note `cudart-*` is the CUDA runtime, a different archive, and both are needed. --- # What actually speeds up decode on a GB10, ranked Decode on this hardware runs at ~80% of its 273 GB/s memory-bandwidth ceiling (see [gb10-bandwidth-ceiling.md](gb10-bandwidth-ceiling.md)), so the only things that help are the ones that **move fewer bytes per token**. This is that list, ordered by measured effect, with the things that look promising and are not. Every figure here is cited to a primary source — an engine PR, vendor documentation, or a paper. Where no primary source exists for a claim, that is stated rather than filled in. ## The structural correction that reorders everything For an MoE model the weight term is **active** parameters only. On a Qwen3-30B-A3B-class model at 32k context, weights are only ~35% of bytes moved once quantised to 4 bits — so the KV cache, not the weights, dominates. At short context the reverse holds. This is why the ranking below is context-dependent and why "quantise the weights harder" stops paying so quickly. ## 1. Pick the architecture first The largest single difference is decided before any flag is set. Gemma-3-27B (5:1 local/global, sliding window 1024) holds **2.91 GB of KV at 32k**; Qwen3-32B holds **8.00 GB**. Nothing downstream recovers a 2.7x difference in the dominant term. ## 2. Quantise the KV cache — the biggest lever, and it is off by default ``` -ctk q8_0 -ctv q4_0 -fa on ``` **+48% throughput for about 0.002 PPL** (llama.cpp PR #7412). Flash attention is required for the quantised-KV path to be taken at all. This is the highest-value line in this document: a measured, large win, available now, that a default configuration does not give you. ## 3. Four-bit weights: already solved — do not re-engineer Use **Q4_K_M** or **MXFP4** and move on. Switching between 4-bit formats is worth about 5%, and one popular choice is worth *less than nothing*: | Qwen3.5-0.8B, same hardware | pp512 | **tg128 (decode)** | |---|---:|---:| | NVFP4 | 25,596 | **388.12** | | Q4_K_M | 38,339 | **521.69** | **Q4_K_M beats NVFP4 at decode by 34%**, despite NVFP4 having native Blackwell support and comparable bits per weight (4.50 vs 4.25 — NVFP4 is the *heavier* format). The reason, from three independent sources: every engine deliberately bypasses the tensor cores below batch 8. NVIDIA's own TensorRT-LLM routes `qinput.shape[0] <= 8` to `cuda_scaled_mm`, whose NVFP4 small-batch kernel is `TILE_M=1` scalar `fma()` with **zero MMA instructions**; llama.cpp caps `MMVQ_MAX_BATCH_SIZE = 8` and decodes NVFP4 through a 16-entry LUT into `dp4a`, an **sm_61 intrinsic**. The one native-FP4 merge reports **prefill only** — there is no decode row in it anywhere, which is itself the tell. ⚠ Also note `Q4_K` the *format* is 4.50 bpw; the widely-quoted ~4.85 is the `Q4_K_M` **recipe**, which promotes `attn_v` and `ffn_down` to Q6_K. They are not comparable numbers. ⚠ **sm_120 FP4 is narrower than "Blackwell has FP4".** Per the PTX ISA, `.kind::mxf4nvf4` and `.kind::mxf4` are supported on **sm_120a and sm_121a only** — excluded from the family-forward `sm_120f` path. And `tcgen05.mma` FP4 kinds are sm_100a/103a/110a only, never sm_120. ## 4. Speculative decoding — the most underused batch-1 lever 3-4x, lossless, and it composes multiplicatively with everything above. It is a bandwidth *amplifier*: the weights are read once and several tokens come out. Note this is about *using* speculation at all, not about tuning how many tokens to draft — see [praecise-method.md](praecise-method.md) for why the tuning question turned out to be second-order. ## Things that look promising and are not **ParoQuant is ~10% slower than AWQ** (2.1x vs 2.3x), not "near-AWQ speed" as its model card states. It buys accuracy at equal bytes, not speed. **No KV eviction method ships.** H2O, SnapKV, Quest and PyramidKV are in neither vLLM nor SGLang. What actually shipped is *trained-in* sparsity — NSA, DeepSeek DSA — which is a property of the model, not a runtime option. Of the post-hoc methods only Quest works on off-the-shelf dense models (2.23x attention, 7.03x end-to-end), and it is a build-it-yourself item. **Activation sparsity is the wrong lever on fast hardware.** TEAL measures 1.53x/1.80x on an A6000 but only 1.25x/1.40x on an A100 — the speedup *shrinks* as the machine gets faster, because it exploits bandwidth slack that a saturated machine does not have. **Host-side optimisation has near-zero return here.** Every published figure is from host-bound H100 serving. GB10 decode at 22.7-38.7 tok/s gives a **~44 ms step budget**, against which detokenisation is under 0.01%. Measure whether the workload is CPU-launch-bound before adopting any of it — at batch 1 on this machine it is not. ## The honest gap **No primary source anywhere gives a PTQ accuracy table comparing NVFP4, MXFP4, AWQ, GPTQ, Q4_K_M and IQ4_XS on a shared model and benchmark.** NVIDIA's only MXFP4 comparison is *pretraining* (8B, 1T tokens, 36% more tokens needed). Anyone ranking these formats for post-training quantisation quality is extrapolating — including sources that present such rankings confidently. The throughput numbers above are measured; a quality ranking is not available. --- # The GB10 decode ceiling **Decode on a DGX Spark is bandwidth-bound and already running at ~80% of peak. There is no software win waiting.** This is the governing constraint on anything Praecise Engine does on this hardware: it bounds every host-side optimisation to the ~20% that is not already saturated, and should be established before any such work is planned. ## The derivation Llama-2-7B Q4_0 weights are 3.56 GiB = 3.82 GB. Decode reads every weight once per token, so tokens/sec × bytes/token is achieved bandwidth: ``` 3.82 GB/token × 57.21 tok/s = 218.7 GB/s 218.7 / 273 GB/s peak = 80.1% ``` Same benchmark, same model, same quantisation, other hardware: | Platform | Peak GB/s | tg t/s | Achieved | Efficiency | |---|---|---|---|---| | **GB10** | 273 | 57.21 | 218.7 | **80.1%** | | M4 Max | 546 | 83.06 | 317.5 | 58.1% | | M3 Ultra | 800 | 92.14 | 352.2 | 44.0% | | M2 Ultra | 800 | 94.27 | 360.3 | 45.0% | GB10 extracts the **highest** fraction of peak in the group. A machine at 44% of peak has software headroom; one at 80% does not. **Independent cross-check.** Applying the same figure to a different model: 218.7 ÷ 60.57 tok/s = 3.61 GB/token on gpt-oss-120b. Over 5.1B active parameters that is **5.66 bits/parameter** — exactly what MXFP4 experts plus higher-precision attention should cost. The model predicts a measurement it was not fitted to, which is the reason to trust it. ⚠ No published STREAM/BabelStream microbenchmark for GB10 exists, so the 80% is inferred from decode rather than measured directly. For this purpose that is arguably the better metric — it is the bandwidth actually reachable by the workload — but it is an inference, not a datasheet number. ## What follows **Ceilings at 219 GB/s achieved:** a 30B Q4 model tops out around 13 tok/s, a 70B Q4 around 5.5 tok/s. Those are arithmetic, not engineering targets. **Arithmetic intensity.** Decode sits at ≈1 FLOP/byte. The FP4 ridge point is 1832 FLOP/byte and the BF16 ridge is 458, so **less than 1% of the advertised FP4 PFLOP is reachable at batch 1**, and batch ~229 would be needed to reach even the BF16 ridge. The compute is not the constraint and mostly cannot be used. **Flash attention buys prefill, not decode**: measured +20% on pp512 and **zero** on tg128, which is what a bandwidth ceiling predicts. **Relative position:** GB10 is 2.08× an M3 Ultra on prefill and 0.62× on decode — it is a prefill machine. Against discrete parts on gpt-oss-20b: RTX 5090 205 t/s, RTX Pro 6000 215 t/s, Spark ~50 t/s. ## Levers that remain, in order 1. **Move fewer bytes per token.** The only lever that moves the ceiling itself. 4-bit is the practical floor: 465+ pretraining runs put the compute-optimal point at 7-8 bits and warn that below 4 bits model size must grow more than 4× to compensate. 4-bit PTQ is effectively solved (QuaRot W4A4KV4 on LLaMA2-70B: ≤0.47 PPL loss). 2. **Remove the KV cache.** Linear and hybrid attention is the strongest decode lever available, because it deletes bytes rather than exploiting slack — so the win *grows* with context and *survives* batching. Kimi Linear reports "up to 6 times decoding throughput for a 1M context" and "reducing KV cache usage by up to 75%"; Qwen3-Next-80B-A3B claims "10 times inference throughput for context over 32K". 3. **Raise batch size.** The only way to reach the compute at all. **Activation sparsity is the wrong lever here**, and its own data says so: TEAL measures 1.53×/1.80× on an A6000 but only 1.25×/1.40× on an A100 — the speedup *shrinks on faster hardware*, because it exploits bandwidth slack that a saturated machine does not have. Its authors concede batch scaling "is a limitation of most activation sparsity work" (per-layer sparsity falls 60% → 38% at batch 4). The 4-5× claims in this area are all CPU/mobile/offload settings. **CPU+GPU cooperation is a capacity mechanism here, not a speed one.** Every offload system that wins arbitrages fast-small VRAM against slow-large DRAM across a narrow PCIe link; GB10 has none of those three (`integrated=1`, `pageableMemoryAccess=1` — genuinely coherent, verified from device properties). Fiddler's scheduling rule compares `cpu_lat` against `gpu_lat + transfer_lat`; set transfer to zero at equal bandwidth and the GPU wins essentially always. Measured on the closest analogue, Strix Halo: moving only `tok_embd` to CPU cost **41.33 → 47.24 t/s, a 14% loss**, because CPU and GPU share the same DDR. ## What this means for this engine **Allocation policy cannot beat a bandwidth wall.** Speculative decoding is a bandwidth *amplifier* — it reads the weights once and emits several tokens — so it remains the right family of technique. But tuning *how many* tokens to draft is a second-order effect on a machine already at 80% of peak, and that is the honest frame for PRAX's measured gains. The first-order work on this hardware is quantisation and KV-cache removal, both of which are properties of the **artefact and the architecture**, not of the serving layer. ⚠ **sm_121 has a real software gap** worth knowing before blaming hardware: six or more open vLLM bugs where frameworks gate on Blackwell family 100 or 120 and sm_121 falls between — MXFP8 and NVFP4 MoE silently falling back to Marlin, `support_deep_gemm()` returning true then hard-faulting, launch failures after 1.5-3 days with **zero Xid** (software, not hardware). Also no prebuilt wheels for `mamba` or `causal-conv1d` on sm_121 — which is awkward, since the hybrid models are the best decode lever and the hardest to build here. ## Attention on sm_121: most of the modern stack is absent Worth knowing before attributing slow decode to the hardware. On SM120/121 vLLM dispatches **FlashInfer -> XQA**; the newer paths are simply unavailable: - **FA3** is SM90-only. - **TRTLLM-Gen** is gated on `major == 10`, so SM100 only. - **CUTLASS FMHA** excluded; **FlashMLA** unusable on sm_121. - **FA4** is released but targets sm_90/100/103 — its SM120 path is a "graceful SplitKV fallback", not a real kernel. Same pattern as the MXFP8/NVFP4 MoE fallbacks: frameworks gate on Blackwell family 100 or 120, and sm_121 falls between. ⚠ **Operational trap.** A FlashInfer cubin version mismatch crash-loops vLLM >= 0.27 on ARM64. Known-good pairing is **vLLM 0.25.1 + flashinfer 0.6.13**. ### Why this does not change the ranking Attention is a small share of decode bytes in the regime that matters here. The crossover context — where KV traffic overtakes weight traffic — is far above typical use at low batch: | Model | B=1 | B=4 | B=8 | |---|---|---|---| | 8B BF16 | 122k | 30.5k | 15.3k | | 8B Q4 | 34.2k | 8.5k | 4.3k | | 70B Q4 | 120k | 29.9k | 15.0k | Below 8k context at batch 1-8, attention is **1.7-6.3% of decode bytes**. Note the Q4 row though: quantising weights *pulls the crossover in* to 4-8k at batch 4-8, so attention starts to matter once the quantisation win has been taken — the levers interact. **Sparse attention splits cleanly on whether it needs retraining.** NSA, DeepSeek DSA and MiniMax MSA all require training from scratch. Only **Quest** works on off-the-shelf dense models (2.23x attention, 7.03x end-to-end) — but it is integrated into neither vLLM nor SGLang, so it is a build-it-yourself item. ⚠ The crossover table is arithmetic from published specs, not a measured GB10 benchmark; no published GB10 decode breakdown exists. The direction is corroborated independently (NSA's arithmetic-intensity argument, and vAttention finding paged and non-paged decode latency similar), but validate before planning against the exact numbers. ## The cheap experiment that would settle the last open question Sweep `--n-cpu-moe` from 0 on gpt-oss-120b, and separately measure whether the GPU alone saturates 273 GB/s. If it saturates, cooperative CPU+GPU execution cannot win by construction. If there is headroom, that is a real and unpublished result. **No published measurement of CPU+GPU cooperative inference on coherent memory exists in either direction** — the absence is itself a finding. --- # Acceleration research — what Praecise Engine should implement Surveyed August 2026 against llama.cpp, vLLM, SGLang and unsloth, with a November 2026 addendum at the end. Every claim below was checked against source, release notes or a paper at the time of writing; version-specific numbers are labelled as such, because several of them have already moved once and will move again. Praecise Engine is a layer, not a runtime. unsloth is the same shape — it sits above the backends and makes them faster — and is the standard of craft this engine aims at: bit-exact optimizations rather than approximations, gradients and reductions hand-derived so results can be written into buffers that already exist, and honest measurement (they once published a correction because their tokens/sec had been counting startup time). llama.cpp, vLLM and SGLang are backends to integrate and optimize for. That framing decides what belongs here. An optimization belongs in Praecise Engine when it is expressible above a backend; it belongs in a backend adapter when it needs that backend's kernels or memory layout. ## What we already have - Speculative decoding: self-speculative block (DFlash) and MTP paths. - Per-architecture matmul tuning; multi-arch CUDA binary, Turing -> Blackwell. - Architecture coverage including muse-glimmer and qwen4exp. - A build-time guard refusing a CUDA build the environment would silently make CPU-only. ## Tier 1 — portable, and worth building here These need no backend's internals, so they are the engine layer's natural work. ### 1. Suffix / n-gram self-speculation The cheapest real speedup in the field. A suffix tree over the prompt plus prior generations, frequency-ranked, proposing continuations. No draft model, no extra weights, no GPU memory — pure CPU token-ID matching. llama.cpp's `ngram-mod` (shared hash pool, constant memory, variable draft length) documents **0.703 acceptance**, against 0.576 for the simpler variant. vLLM ships the same idea as suffix decoding, defaulting to a *maximum* of 32 speculative tokens chosen adaptively per request per step. Two properties make this the highest value-per-effort item for a layer: it needs nothing from the backend but token history, and — uniquely — the index can span requests and sessions. Agentic loops, code editing and RL rollouts repeat themselves heavily, and a cross-session suffix index is something no single runtime is positioned to build. ### 2. Speculation as a portfolio, with honest economics Speculative decoding is not one feature to switch on. Choose per model, per hardware, per load: - Published speedups are almost always quoted at concurrency 1 and decay hard under load. vLLM's P-EAGLE measures 1.55x at c=1 and **1.05x at c=64** on MT-Bench. Speculation trades FLOPs for latency, and a loaded server has none spare. - **MoE inverts the economics.** Parallel verification activates far more experts than single-token decode. llama.cpp measures EAGLE3 at 3.28x on dense Llama-3.1-8B but **1.08x on GPT-OSS-120B**. - MTP is a latency win that costs throughput: it forces `n_parallel=1` and drops prefill to roughly half, because hidden states move device->host. - The field has moved past tree verification. DFlash verifies a linear block, DSpark schedules a linear window, and SGLang's CPU/GPU overlap scheduler supports only `topk=1`. Designing for trees aims at a receding target. The portable part of the frontier is a *scheduling policy*: DSpark sizes each verify window from the draft's own calibrated confidence against a pre-profiled steps-per-second curve. That is implementable above any backend that exposes per-position draft confidence. Its paper also flags a trap worth respecting — a scheduler searching without causal early stopping biases the output distribution toward tokens that happen to trigger longer verifications. Causal stopping is load-bearing for correctness, not just speed. ### 3. Per-hardware x per-quant-type dispatch The single global `MMVQ_MAX_BATCH_SIZE = 8` left up to **49%** on the table. llama.cpp replaced it with an empirically swept table: RTX 5090 crosses at batch 5 for Q2_K..Q5_K and 7 for Q6_K; RTX 4090 at 4/6/7; DGX Spark only Q2_K at 6; legacy quants stay at 8. Two things generalize. The methodology: sweep with **clocks pinned** (`llama-bench -p 1..8 -n 0 -embd 1 -r 50`), because thermal drift is larger than the effect being measured. And the classification, which matters more than any single number: **MMVQ is bandwidth-bound and wants occupancy; MMQ is register-pressure-bound and wants per-block resources.** The same occupancy change measured +3-6% on one and **-30%** on the other. Never apply one policy globally. ### 4. Evaluate quantization by trajectory divergence Perplexity and single-token top-1 both under-detect damage that compounds over a generation. The stronger metric now in use is KL divergence over a multi-token greedy decode against the unquantized reference — unsloth scores 300 held-out examples over 32 tokens. Any quantization decision this engine makes should be justified that way rather than by perplexity. Per-tensor bit allocation itself is a *packaging* decision, not a runtime one: `llama-quantize --tensor-type` takes regexes, so a scheme is expressed as data and costs nothing at inference time. ### 5. Structured output: pick the backend before dispatch, not after The mechanism worth adopting is llguidance's lexer/parser split. GBNF has no lexer; llguidance runs a regex lexer in front of the CFG parser, and because there are ~10x fewer lexemes than bytes and tokens tend to align with lexemes, **the CFG parser is engaged on under 0.5% of tokens.** Measured mask cost is ~50us mean, 0.5ms p99 on a 128k vocab. Backend selection should be a *schema inspection* up front, not an exception caught downstream. vLLM discovers xgrammar cannot handle a schema by catching a validation error; the predicate is a pure function and can be run first. This matters beyond tidiness: xgrammar **silently drops `minLength`/`maxLength`** when combined with `pattern`/`format`, so output can violate the schema with no error raised. Grammar failures must degrade, never abort. A grammar that gives up should be reported once, not once per token. ## Tier 2 — backend-adapter work Real wins, but they need a backend's internals, so they belong in adapters. - **Paged KV.** llama.cpp's unified cache preallocates `n_ctx * n_seq_max`, so concurrency is bounded by *maximum* context rather than average. A paged prototype reaches 247 concurrent sequences against 25 for unified (2.5x aggregate throughput) at equal low-concurrency performance. Its user-facing defrag knob is already deprecated in favour of context checkpoints. - **KV cache quantization.** FP8 KV measures +14.9% throughput and -14.8% median ITL on Llama-3.1-8B/H100 — but the break-even context length is *version-specific* (24,889 tokens on one release, 7,010 on a later one; 741,565 vs 22,109 on another model). Two gates: head_dim=256 makes TTFT ~1.6x slower, and on hybrid models skipping sliding-window layers beats quantizing everything. - **Flash attention** is now tri-state and defaults to `auto`. The trap is that quantized K/V combinations are only compiled with `GGML_CUDA_FA_ALL_QUANTS=ON`; without it an unsupported combination silently falls off the FA path. - **Blackwell FP4.** The generic NVFP4 MMQ kernel gives +118% to +191% on prefill but is **flat (~0.99x) on token generation** — a prefill win, not a decode one. ## Correctness traps to test for explicitly Cheap to check, and each one produces plausible-looking bad output rather than an error: - **BOS handling.** Adding or omitting the start-of-sequence token is the single most reported cause of gibberish, endless generation and repetition across engines. Assert it rather than trusting the template. - **Untrained tokens.** Reserved/special tokens ship with garbage embeddings in some base models; constrained decoding can *force* sampling into exactly those. - **Per-model numerical quirks** — logit scaling, logit softcapping — silently corrupt output when a runtime omits them. - **Tokenizer bugs live in the runtime, not the artifact.** At least one Gemma 4 tokenizer fix was C++-only and needed no GGUF regeneration, so "re-download the model" is often the wrong remedy. - **A denominator correct per-microbatch is wrong across microbatches of unequal token count.** The same class of bug appears in any per-sequence-averaged metric under ragged batching. ## What this implies for the roadmap 1. Suffix/n-gram self-speculation, cross-session — highest value per unit effort, **implemented** as `praecise_runtime::ngram::NgramCache` (backend-agnostic; still to wire into the decode loop and measure against the 0.703 reference), and structurally ours rather than any backend's. 2. A speculation policy layer: pick method and verify-window per model, hardware and load; refuse to speculate where it does not pay (MoE under concurrency). 3. Per-hardware x per-quant dispatch tables, swept with pinned clocks. 4. A trajectory-divergence harness, so quantization and speculation claims are measured the way they actually degrade. 5. Schema-inspecting structured-output routing with a shared compile cache. ## Where the frontier converged (verified August 2026) The models this engine has to serve are no longer a zoo of unrelated designs. Four labs independently shipped the same primitives, which is what makes an acceleration layer tractable at all. **The common design.** DeepSeek V4, GLM-5.3-Flash, Kimi K3 and qwen4exp all run MoE with aux-loss-free (`noaux_tc`) balancing plus shared experts, compressed latent attention, a learned indexer selecting a sparse top-k, and Sinkhorn- projected hyper-connections. The hyper-connection hyperparameters are *byte-identical* across DeepSeek V4 and GLM-5.3-Flash (`hc_mult: 4`, `hc_sinkhorn_iters: 20`, `hc_eps: 1e-06`), and llama.cpp reuses DeepSeek V4's `HC_PRE` kernel for Kimi K3. One kernel serves four architectures. **A 3:1 hybrid of linear and full attention is now the mainstream layout** -- qwen4exp, GLM-5.3-Flash and Kimi K3 all use it. Gemma 4 (5:1) and muse-glimmer (3:1) alternate sliding and full instead. So the engine needs exactly three attention regimes, not one per model: hybrid linear+full, sliding/full alternation, and indexer-based block-sparse. This is worth stating plainly about our own work: **qwen4exp's hybrid memory, block-sparse QSA indexer and gated hyper-connection residual are the convergent frontier design, not an outlier.** That is a reason to invest in the shared primitives rather than treat the architecture as bespoke. **Speculation has four shapes** to support: MTP heads in the model, dedicated draft models, DSpark (confidence-scheduled), and DFlash (block diffusion). Note Kimi K3 ships `num_nextn_predict_layers: 0` -- no MTP head at all -- and relies on DSpark. An engine that assumes MTP is available will simply have nothing to speculate with. ### Architecture traps to test against Each of these degrades silently rather than erroring: - **NoPE is spreading.** GLM-5.3-Flash and Kimi K3 have *no RoPE in the text tower* (`mla_use_nope: true`, `qk_rope_head_dim: 0`); position comes from the linear-attention recurrence. muse-glimmer drops RoPE on global layers only. Code that assumes RoPE everywhere mis-serves two frontier families. - **Keep selection machinery out of the quantized path.** Every one of these models quantizes routed experts to FP4/MXFP4 while holding indexers, gates and hyper-connection mixers at higher precision. Quantizing an indexer corrupts top-k selection, which degrades quality without any error -- and that applies directly to qwen4exp's QSA indexer. - **Frontier models increasingly ship no working Jinja chat template.** DeepSeek V4 and Kimi K3 have no `chat_template` at all and need custom encoders; DeepSeeks format uses full-width characters in its role markers. This is the same class of failure as the BOS trap below, and it is why `force_pure_content` exists as an escape hatch. - **Some models are not bit-exact between cached and fresh runs** (an MXFP4 kernel constraint on DeepSeek V4). Equality assertions in tests will flake; compare scores, not tokens. ## Implementation language: what the evidence actually says Worth recording, because the intuition here is wrong in both directions. **Host overhead is real and large.** Profiling vLLM on an H100 with Llama-3-8B found GPU execution was only **38%** of step time -- scheduling 29%, API server 33%. An independent study measured scheduling at up to **~50%** of end-to-end latency. "The kernels dominate, so the host language does not matter" is false. **But no one solved it by changing language.** vLLM stayed in Python and moved input preparation onto the GPU (Model Runner V2, **+56%** throughput); SGLang's overlap scheduler runs one batch ahead, worth **10-20%**; CUDA graphs remove the host from the launch loop entirely. The fix is always *taking work off the critical path*, not making that work faster. Two natural experiments already ran, and both cut against a compiled orchestration layer. NVIDIA **deleted** TensorRT-LLM's C++ backend ("PyTorch is now the sole execution backend", v1.2) for developer efficiency. And TGI -- the Rust-router precedent -- was **archived** in March 2026, pointing users at vLLM/SGLang/llama.cpp. Rust did not save it; failing to keep pace with model architectures did. That is the real cost to weigh: EAGLE-3, P-EAGLE, DFlash and DSpark all arrive as Python/Triton PRs against vLLM or SGLang. There is **no non-Python adoption path**. A compiled layer reimplements each by hand and permanently trails. **Where a compiled layer genuinely wins is embedding.** Praecise Engine is a library that loads inside someone else's process, and there Python's single interpreter, GIL and dependency weight are disqualifying -- vLLM's own fix for GIL contention was to move its API server into a *separate process*. Rust gives a `cdylib` with no runtime to initialise. So the split this engine already has is the defensible one: Rust for the embeddable layer, CUDA C++ for kernels. The conclusion to carry forward is not about language at all -- it is that **host work must come off the critical path**, by preparing inputs on the GPU and overlapping scheduling with compute. Note also that no controlled benchmark of a Rust orchestration layer against a Python one over identical kernels appears to exist publicly; anyone claiming a decisive language win in either direction is extrapolating. ## Kernel strategy is set by batch shape, not by hardware The assumption worth discarding first: "batch-1 decode leaves tensor cores idle, so the vector path always wins." That is true of some engines and false of others, and the difference is not the GPU. Verified by reading the kernels at HEAD (2026-08-27): | Codebase | Batch-1 path | |---|---| | llama.cpp | dp4a SIMT GEMV — `mmvq.cu` contains no MMA at all | | FlashInfer | scalar FMA + warp shuffle — `decode.cuh` has none, `prefill.cuh` has 40 | | vLLM | **stays on tensor cores** — no GEMV kernel, no `dp4a` anywhere in `csrc/` | vLLM's Marlin specializes tiles for small M but keeps the arithmetic on `mma.sync.aligned.m16n8k16`, adapting through split-K to fill the SMs rather than leaving the tensor cores. Its AllSpark W8A16 path inverts the naive expectation outright: small M runs a tensor-core kernel, while large M (>1024) falls back to cuBLAS. **The dividing line is whether a deployment ever presents M=1 to the GEMM.** llama.cpp's steady state is single-user local decode, which is M=1 exactly. vLLM's is a continuous-batching server holding dozens of sequences, so it has little reason to keep a true GEMV path at all. That is a decision this engine has to make explicitly rather than inherit. A batching server should follow vLLM's precedent — tensor cores plus split-K — even at low nominal batch, because the effective M is the number of concurrent sequences, not one. Two more measured points worth keeping: - `MMQ_DP4A_MAX_BATCH_SIZE = 64`: even on hardware with FP16 tensor cores, llama.cpp prefers integer SIMT below batch 64. - Tensor-core availability *shrinks* the vector regime rather than widening it (`mmvf.cu`): Ampere and later use the vector path only at `ne11 == 1`, Ada up to 4, and hardware without tensor cores up to 8. The per-quant tables carry comments naming the GPU they were tuned on — these are measurements, not derivations, which is the standard any tuning added here should meet. ### Two corrections to received wisdom **H100 at batch 1 is launch-bound, not bandwidth-bound** — roughly 26% of peak. CUDA graphs are worth ~1.26x there and nothing on an L4. Buying HBM bandwidth does not buy batch-1 latency, so "bandwidth ÷ bytes-per-token" is a ceiling, not a prediction; on a fast GPU at batch 1 the host and launch path bind first. This is the same conclusion the host-overhead numbers reach from the other direction. **Halve NVIDIA's headline tensor figures unless they say dense.** The GB200 page states the convention in plain text: the large number is sparse, and dense is half of it. ## November 2026: what moved, and what did not Surveyed 2026-11-13 against llama.cpp 0.7.0 to 0.9.0, vLLM, SGLang and the papers named below. The question was narrow: is there a technique, new since the August survey, that makes decode faster and that this engine can adopt. **No new batch-1 decode kernel was found.** Nothing published since August reads fewer bytes per token from a 4-bit model without losing quality. The levers that pay are still the ones this document already ranks: more tokens per pass (speculation), and a model class that emits many tokens per pass (diffusion). The upstream release notes for 0.8.0 and 0.9.0 carry no benchmark figures, so every gain attributed to them below has to be measured here. Each entry says which of two things it does, because decode here is memory-bandwidth-bound: **fewer bytes per token**, **more tokens per pass**, or neither. | # | Technique | Published gain | Lossless | Where it lands here | Bandwidth effect | |---|---|---|---|---|---| | 1 | llama.cpp 0.8.0 and 0.9.0 (ggml 0.28, 0.29): CUDA chunked graph launches, faster low-bit mat-vec, Metal MXFP4 mat-vec, Vulkan split-K on RDNA3/4, Arm SVE Q8_0/Q6_K/MXFP4, SYCL flash attention over a quantised KV cache, the n-gram cache saved to a file | None published | Yes | Linked route; the fork is synced with 0.9.0, built and tested on the CPU only ([silicon-coverage.md](silicon-coverage.md)) | Neither; launch cost and kernel efficiency | | 2 | LiLiCorr: a two-layer transformer rescoring a top-K lattice of DFlash drafts | Acceptance length +7 to +19% over DFlash at about 3% overhead per block (arXiv 2608.20530, abstract; models and hardware not stated there) | ⚠ Not addressed in the abstract | The speculation policy layer; needs a trained checkpoint per target model | More tokens per pass | | 3 | DFlash drafts in SGLang Spec V2 | Over 4.3x baseline and 1.5x MTP on Qwen 3.5 397B-A17B, HumanEval, concurrency 1, 8xB200; the overlap scheduler alone takes Qwen3-8B from about 11.4 to 15.3 ktok/s at concurrency 32 on one B200 | Yes, under greedy verification | Hosted SGLang route, a configuration change | More tokens per pass | | 4 | MoE expert cache on the GPU (`--moe-cache-mib`), for models that do not fit in VRAM | 25.0 to about 40 t/s on an RTX 4090 and 30.8 to 67.8 t/s on an RTX 5090, Qwen3.8-Flash-Next Q4_0 | ⚠ Bit-identity not stated | The runtime sets the cache size on discrete GPUs | Neither; it removes the PCIe path | | 5 | Diffusion language models (DiffusionGemma 26B-A3.8B, Apache 2.0) | ⚠ 1,008 tok/s on H100 and 1,288 on H200, FP8, batch 1, in vLLM (trade press; the model card was not reachable) | No: a different model, and its maker recommends the autoregressive one for quality | Hosted vLLM route. The linked route has only a draft upstream pull request | More tokens per pass; the largest such lever | | 6 | ZipServ (ASPLOS 2026): lossless weight compression decoded inside the matrix product | Up to 30% smaller weights, 1.22x over vLLM on average at batch 8 and 32, RTX 4090 and L40S | Yes, bit-exact | Hosted BF16 serving only; a port, not a setting | **Fewer bytes per token**, the only new lossless technique found that does this | | 7 | CUDA graphs kept on for MoE (upstream pull request 26802) | 1.13x at 16 sequences, 1.08x at 32, flat at batch 1 (Qwen3-30B-A3B Q4_K_M, RTX 5090, secondary write-up) | Yes | Already in the fork | Neither | | 8 | Arm SME2 matrix products for Q4_K and Q6_K through KleidiAI | None in the release notes | Yes | `kleidiai` feature; a prefill gain is the expectation | Neither | **Negative results**, recorded so they are not tried twice: - **The MoE expert cache has made a machine slower.** One report on two Radeon R9700 cards under Vulkan fell from 23 to 18.1 t/s. On unified memory (GB10, Apple, Strix Halo) no expert is host-resident, so the cache should have nothing to do; that is an inference from the design, not a measurement. - **Diffusion decoding is poor off NVIDIA today.** The draft upstream pull request reports about 27 tok/s on a Radeon R9700 under Vulkan and no Metal result. llama.cpp 0.9.0 runs the one diffusion model it loads in autoregressive mode only. - **ZipServ has no batch-1 result**, covers BF16 weights only, and its authors report it can lose to cuBLAS on A100 and H800 and to lossy FP8. - **Native 4-bit floating point is still a prefill feature upstream**, not a decode one. - **Lossy KV eviction and 3-bit KV quantisation** were not adopted. The eviction work reports 1.69x over vLLM but gives up accuracy, and it is a preprint read through an aggregator. ⚠ The 3-bit KV reports were not opened. - **Nothing new for early exit, ternary weights or sub-4-bit weights** with a measured 2026 decode figure was found. - ⚠ **Matrix extensions (AMX, SME, Qualcomm's) for decode** were not verified; the one figure seen, about 1.05x at four threads, came from a search snippet. It agrees with the bandwidth ceiling this repository measures. **Two leads not yet read to the source.** llama.cpp 0.7.0 lists "MXFP4 MoE with arithmetic decoding and reordering" for SYCL; if that is entropy-coded weights, it is a shipped reduction in bytes per token on Intel, and worth one reading of the pull request. ⚠ OpenVINO GenAI is said to have extended its speculative decoding to CPU, GPU and NPU; the release notes were not opened. ### What this changes in the roadmap 1. The fork is synced with llama.cpp 1.0.0 and built on the CPU only; no GPU backend has been compiled against it. Measure decode before and after on the GB10. Until that measurement exists, no gain is claimed for it. 2. Let the speculation policy layer choose a DFlash corrector where a checkpoint exists, and refuse where none does. 3. Size the MoE expert cache from the hardware layer: on for a discrete GPU whose VRAM is smaller than the model, off on unified memory. 4. Treat a diffusion model as a hosted-route model until the linked route can run one. `BatchEngine` still refuses them, which is correct. Sources: llama.cpp release notes for 0.6.0 to 0.9.0 (read through a release mirror); upstream pull requests 29887, 24423 and 26802; arXiv 2608.20530, 2603.17435 and 2606.16332; the vLLM speculative decoding documentation; the SGLang DFlash write-up of 2026-06-15. --- # qwen4exp: provenance, and how to get it properly **qwen4exp is upstream llama.cpp's work, not ours.** This note exists because a modified copy of it sat uncommitted in the vendored fork, and committing that to `Praecise/llama.cpp` would have published someone else's architecture implementation under our name with no attribution. ## Who wrote it ``` commit 6c84c7d5d8833c6e0df69628f75a0f599797934e author Daniel Han date 2026-08-27T19:32:31Z title model: add Qwen3.8-Flash-Next (qwen4exp) (#27742) repo ggml-org/llama.cpp ``` It is the only commit touching `src/models/qwen4exp.cpp`, so the whole implementation -- the architecture, the hybrid memory index, the conversion script -- arrived in that one PR. Merged upstream the day before this work started. ## How to tell, in two minutes The check that settles it, and that should have been run first: ```sh curl -o /dev/null -w '%{http_code}\n' \ https://raw.githubusercontent.com/ggml-org/llama.cpp/master/src/models/qwen4exp.cpp ``` `200` means upstream has it. A local copy of a file upstream already ships is **vendoring**, not authorship, however much local editing it has accumulated. The local copy was 1,199 lines against upstream's 1,199 -- identical length, differing by ~440 lines of reordering and local edits. Same file, modified. ## The right way to get it **Do not commit the local copy to the fork.** Take it from a newer upstream instead, so upstream's authorship travels with the code. qwen4exp lands between build tags **b10658** (absent) and **b10660** (present): | tag | has `src/models/qwen4exp.cpp` | |---|---| | b10650, b10652, b10654, b10656, b10658 | no | | **b10660**, b10666, b10667 | **yes** | ⚠ **A rebase does not work here, despite being the obvious answer.** `Praecise/llama.cpp` shares **no history with upstream** — `git merge-base` against b10660 returns nothing. The repo has two commits, and the older one is a **squashed import with no parent**: 3,396 files and 1,435,194 insertions in a single root commit. There is nothing to rebase onto. So the options are: **A. Re-import from b10660+.** Take upstream at that tag, re-apply the local changes listed in `MODIFICATIONS.md`, commit under the correct identity. Same shape as today, current, and qwen4exp arrives with it. **B. Re-fork properly.** Clone upstream with its real history and put the local commits on top, so `git log` shows upstream's authorship rather than only the tree doing so. More work, and it moves the submodule pin. ## On the licence, which is the thing that actually matters **The current arrangement is not a violation.** The import preserves upstream's `LICENSE`, `AUTHORS` and `README.md`, and `MODIFICATIONS.md` states it directly: > "This is **not** stock upstream llama.cpp... Upstream: > https://github.com/ggml-org/llama.cpp — Copyright (c) 2023-2026 The ggml > authors, MIT License. Those terms continue to govern this directory... The > Apache-2.0 licence applied to Praecise Engine's own crates does not relicense > this code." MIT asks that the copyright notice travel with the code, and it does. The commit *metadata* is a poor record of provenance — a squashed import authored locally reads as though the code were written here — but the attribution a reader needs is present in the tree. ## What the fork legitimately owns One of its two commits, and it is real work: ``` dddf1d9 Halve the MMVQ warps per block on Blackwell ``` A measured decode-kernel tuning -- two warps per block instead of four, so more blocks stay resident on an SM and more memory requests are in flight to hide each other's latency. Decode is bandwidth-bound, so latency hiding is what helps. That belongs to us and should survive the rebase. ⚠ Its author identity differs from the repository's convention of `Hilal Agil ` for both author and committer. Worth fixing while rebasing, since the history is being rewritten anyway. ## The general rule Before committing vendored code to a fork under our name, check whether upstream already ships it. A fork exists to carry *our* changes on top of theirs, not to re-author theirs. When the answer is "upstream has this", the correct action is always to move the base forward rather than to commit the copy.