Running several engines
An application can serve several models at once, each on the engine that suits
it: llama.cpp in its own process, and any hosted backend in a runtime host.
praecise_host::Engines is the one place models are assigned to engines:
- One engine per model. Every load claims the model id first; a claim held
by another engine, or a second host for the same model, is refused with
Error::Conflict. A model is never in memory twice. - One memory budget. Each claim reserves the model's memory from a
Budgetshared by every engine: the application's own ledger, or aFixedBudget. - Confined, not trusted. A runtime host is a child process the application
starts by re-running its own executable as the confinement launcher (call
praecise_host::sandbox::enter_if_requested()first thing inmain). It gets new user, network, mount, IPC and PID namespaces, a Landlock ruleset (read-only engine environment and model files, one scratch directory, the GPU device nodes), a seccomp filter and resource limits. It has no network: no interface but an optional loopback with no route out, and no internet socket unless loopback is granted. It speaks to the application only over its standard streams, one JSON object per line, with request ids so an engine that batches sees every outstanding request. - Pinned environments. An
EngineEnvis a directory holding the engine's program, packages and protocol adapter, pinned by the SHA-256 of its whole tree (praecise_host::engine::digest). The digest is checked before every start; an environment that changed is refused, never run. - Supervised. A host that exits fails the requests it was serving and is restarted in the background from the same verified environment, while every other engine keeps serving. After three failures in a row it is left stopped and reported, until the model is released and served again.
- Accelerator or refusal. On a machine with an accelerator of any vendor, an engine that cannot use it refuses to load rather than falling back to the CPU. A host is given the device nodes of every vendor (NVIDIA, AMD, the kernel's DRM and compute-accelerator classes, Ascend, Neuron and the rest), the vendor runtime directories, and the device-selection variables the operator set.
The hosted backends are driven through their offline Python engine APIs (vLLM
AsyncLLM, SGLang Engine, TensorRT-LLM LLM, transformers, OpenVINO GenAI
LLMPipeline) by one adapter,
praecise-host/adapters/praecise_adapter.py, which engine::install_adapter
writes into a provisioned environment before it is pinned. No engine opens an
HTTP server. praecise_host::generation maps GenerationConfig and
InferenceResult onto the protocol, so a hosted engine answers with the same
types, token counts and stop reasons as the linked one; a reply without token
counts is refused rather than metered as free.
A kernel that forbids unprivileged user namespaces (for example
kernel.apparmor_restrict_unprivileged_userns=1) refuses the confinement, and
the error names the setting; grant the application's executable userns in
its AppArmor profile rather than lowering the setting for the whole machine.
On Ubuntu that is one profile per executable, for example
/etc/apparmor.d/praecise-job:
abi <abi/4.0>,
include <tunables/global>
profile praecise-job /opt/praecise/bin/praecise-job flags=(unconfined) {
userns,
}followed by apparmor_parser -r /etc/apparmor.d/praecise-job. Where that is
not possible an operator may choose --confinement landlock-only explicitly:
the same Landlock ruleset and syscall filter (no internet sockets, no file
outside the rules), without the namespaces. It is never chosen on its own.
Job engines
Beside the model engines, praecise_host::job runs job engines: a fixed
program with typed parameters, input files in, output files and progress out,
run once to completion. A job engine is a pinned environment (the same
SHA-256 tree digest, checked before every run) and one Python adapter,
praecise-host/adapters/praecise_job.py, compiled into the binary and placed
read-only beside the run's scratch directory. Every run is confined like a
model host: no network, the environment, model caches and inputs read-only,
one scratch directory writable, the GPU device nodes; it is supervised (time
limit, the whole process tree stopped at the end, the last lines of its error
output on failure); on a machine with a GPU a program refuses rather than run
on the CPU. Parameters are checked against each program's declaration (type
and range) before anything starts, and again by the program.
The programs drive NVIDIA's open simulation systems. They are NVIDIA's own
products and run on CUDA; the ones built on torch also run on AMD ROCm, which
presents as cuda:
| Program | System | Out |
|---|---|---|
sionna.radio_map |
Sionna RT | radio_map.npz, radio_map.png |
warp.example |
Warp | the example's USD stage |
newton.example |
Newton | sim.usd |
cosmos.generate |
Cosmos 3 (guardrail forced on) | video.mp4 |
physicsnemo.darcy_fno |
PhysicsNeMo | fno_darcy.pt and its error on unseen fields |
cudaq.ghz, cudaq.maxcut |
CUDA-Q | sampled states, the best cut against the optimum |
boltz.predict |
Boltz-2 | structures, confidence, affinity |
Each writes result.json. The praecise-job binary runs one:
praecise-job programs # the catalogue, with every parameter's type and range
praecise-job digest /opt/envs/cudaq # the digest to pin an environment by
praecise-job run --engines engines.json --program cudaq.ghz --param qubits=24 --out out/engines.json names each engine's environment and pin:
{"engines": {"cudaq": {"root": "/opt/envs/cudaq", "digest": "<sha256>", "read_only": [...], "env": {...}}}}. Progress is printed as JSON lines and
the last line is the result.