Praecise Engine v0.1.0 Read the docs

Documentation · Overview

Why speculative decoding

Single-stream (batch-1) decode is memory-bandwidth-bound: each token streams the model's active weights from memory once, so throughput is bounded by bandwidth ÷ bytes-per-token. The software levers are to move fewer bytes per token (quantization, sparsity) or to emit more tokens per weight-read (speculative decoding). Praecise Engine focuses on the latter — techniques that are largely backend-independent, which is why they belong in a layer rather than a fork.

For what moves the ceiling itself rather than working within it, see docs/what-actually-helps.md. The short version: KV-cache quantization is worth +48% and is off by default.

Edit this page on GitHub · Read it as Markdown

    Type to search. Arrow keys move, Enter opens.