Praecise Engine v0.1.0 Read the docs

Documentation · Using it

Selecting a backend

praecise_runtime::Backend names the runtimes the engine knows about and reports, per backend, what is actually possible:

rust
use praecise_runtime::Backend;

let backend = Backend::parse("vllm")?;   // unknown names error, never default
backend.ensure_available()?;             // refuses rather than silently substituting
let caps = backend.supports();

The distinction that matters is how a backend is reached, because it decides what acceleration is available at all:

Backend Integration Engine-side speculation Structured output
llama.cpp linked (FFI) yes — the engine drives the decode loop yes
vLLM hosted no — the runtime owns its decode loop no
SGLang hosted no no
TensorRT-LLM hosted no no
transformers hosted (CPU or GPU) no no
OpenVINO GenAI hosted (Intel CPU, GPU or NPU) no no
an inference API remote (opt-in) no — the provider owns the decode loop no

A linked backend is compiled in, so the engine can propose a draft block, verify it, and read per-position logits. A hosted backend runs in a runtime host: the engine chooses the model and the sampling parameters, but there is no seam at which to interpose on token-by-token decoding. Speculation against a hosted backend is therefore not merely unimplemented — it is impossible by construction, and those runtimes do their own instead.

A remote backend runs nowhere on this machine: praecise-remote asks an OpenAI-compatible inference API (Groq, Cerebras, SambaNova or any other) and answers in the same types. It is the one route to silicon sold as a service, and the one backend whose prompts leave the machine, so it is opt-in twice: the application has to depend on the crate, and the operator has to name each host in PRAECISE_REMOTE_ALLOW. Runtime hosts still have no network.

rust
use praecise_remote::{Allow, Provider, Remote};

let groq = Remote::open(Provider::Groq.endpoint(model), &Allow::from_env())?;
let reply = groq.generate(&messages, &config, |text| { print!("{text}"); true })?;

An unknown backend name is an error listing the valid ones, never a quiet fallback: a typo that silently ran on a different runtime would invalidate any measurement taken against it, with nothing in the output to show it.

Edit this page on GitHub · Read it as Markdown

    Type to search. Arrow keys move, Enter opens.