What this layer is for, measured
Decode is memory-bandwidth-bound. On a DGX Spark (GB10) it already runs at ~80% of the 273 GB/s peak — the highest fraction of peak of any comparable machine (M4 Max 58%, M3 Ultra 44%). See docs/gb10-bandwidth-ceiling.md.
That constrains what a layer above a backend can honestly claim: everything host-side shares the ~20% that is not already saturated. So this engine concentrates on decisions the backend cannot make for itself, and on avoiding losses rather than manufacturing wins.
Measured on an RTX 5070 Ti with qwen3.5-0.8b and n-gram self-speculation:
| prompt | tok/s | drafted | vs baseline |
|---|---|---|---|
| SQL | 226.3 | 171 | 3.5x |
| code | 94.5 | 73 | 1.5x |
| number list | 63.8 | 0 | — |
| JSON | 65.4 | 0 | — |
| prose | 62–65 | 0 | — |
Speculation is worth 1.5–3.5x on structured text and nothing on prose. The decision of whether to speculate is where the value is; see docs/measured-acceptance.md.