Praecise Engine v0.1.0 Read the docs

Documentation · Using it

Running several engines

An application can serve several models at once, each on the engine that suits it: llama.cpp in its own process, and any hosted backend in a runtime host. praecise_host::Engines is the one place models are assigned to engines:

  • One engine per model. Every load claims the model id first; a claim held by another engine, or a second host for the same model, is refused with Error::Conflict. A model is never in memory twice.
  • One memory budget. Each claim reserves the model's memory from a Budget shared by every engine: the application's own ledger, or a FixedBudget.
  • Confined, not trusted. A runtime host is a child process the application starts by re-running its own executable as the confinement launcher (call praecise_host::sandbox::enter_if_requested() first thing in main). It gets new user, network, mount, IPC and PID namespaces, a Landlock ruleset (read-only engine environment and model files, one scratch directory, the GPU device nodes), a seccomp filter and resource limits. It has no network: no interface but an optional loopback with no route out, and no internet socket unless loopback is granted. It speaks to the application only over its standard streams, one JSON object per line, with request ids so an engine that batches sees every outstanding request.
  • Pinned environments. An EngineEnv is a directory holding the engine's program, packages and protocol adapter, pinned by the SHA-256 of its whole tree (praecise_host::engine::digest). The digest is checked before every start; an environment that changed is refused, never run.
  • Supervised. A host that exits fails the requests it was serving and is restarted in the background from the same verified environment, while every other engine keeps serving. After three failures in a row it is left stopped and reported, until the model is released and served again.
  • Accelerator or refusal. On a machine with an accelerator of any vendor, an engine that cannot use it refuses to load rather than falling back to the CPU. A host is given the device nodes of every vendor (NVIDIA, AMD, the kernel's DRM and compute-accelerator classes, Ascend, Neuron and the rest), the vendor runtime directories, and the device-selection variables the operator set.

The hosted backends are driven through their offline Python engine APIs (vLLM AsyncLLM, SGLang Engine, TensorRT-LLM LLM, transformers, OpenVINO GenAI LLMPipeline) by one adapter, praecise-host/adapters/praecise_adapter.py, which engine::install_adapter writes into a provisioned environment before it is pinned. No engine opens an HTTP server. praecise_host::generation maps GenerationConfig and InferenceResult onto the protocol, so a hosted engine answers with the same types, token counts and stop reasons as the linked one; a reply without token counts is refused rather than metered as free.

A kernel that forbids unprivileged user namespaces (for example kernel.apparmor_restrict_unprivileged_userns=1) refuses the confinement, and the error names the setting; grant the application's executable userns in its AppArmor profile rather than lowering the setting for the whole machine. On Ubuntu that is one profile per executable, for example /etc/apparmor.d/praecise-job:

abi <abi/4.0>,
include <tunables/global>
profile praecise-job /opt/praecise/bin/praecise-job flags=(unconfined) {
  userns,
}

followed by apparmor_parser -r /etc/apparmor.d/praecise-job. Where that is not possible an operator may choose --confinement landlock-only explicitly: the same Landlock ruleset and syscall filter (no internet sockets, no file outside the rules), without the namespaces. It is never chosen on its own.

Job engines

Beside the model engines, praecise_host::job runs job engines: a fixed program with typed parameters, input files in, output files and progress out, run once to completion. A job engine is a pinned environment (the same SHA-256 tree digest, checked before every run) and one Python adapter, praecise-host/adapters/praecise_job.py, compiled into the binary and placed read-only beside the run's scratch directory. Every run is confined like a model host: no network, the environment, model caches and inputs read-only, one scratch directory writable, the GPU device nodes; it is supervised (time limit, the whole process tree stopped at the end, the last lines of its error output on failure); on a machine with a GPU a program refuses rather than run on the CPU. Parameters are checked against each program's declaration (type and range) before anything starts, and again by the program.

The programs drive NVIDIA's open simulation systems. They are NVIDIA's own products and run on CUDA; the ones built on torch also run on AMD ROCm, which presents as cuda:

Program System Out
sionna.radio_map Sionna RT radio_map.npz, radio_map.png
warp.example Warp the example's USD stage
newton.example Newton sim.usd
cosmos.generate Cosmos 3 (guardrail forced on) video.mp4
physicsnemo.darcy_fno PhysicsNeMo fno_darcy.pt and its error on unseen fields
cudaq.ghz, cudaq.maxcut CUDA-Q sampled states, the best cut against the optimum
boltz.predict Boltz-2 structures, confidence, affinity

Each writes result.json. The praecise-job binary runs one:

praecise-job programs                      # the catalogue, with every parameter's type and range
praecise-job digest /opt/envs/cudaq        # the digest to pin an environment by
praecise-job run --engines engines.json --program cudaq.ghz --param qubits=24 --out out/

engines.json names each engine's environment and pin: {"engines": {"cudaq": {"root": "/opt/envs/cudaq", "digest": "<sha256>", "read_only": [...], "env": {...}}}}. Progress is printed as JSON lines and the last line is the result.

Edit this page on GitHub · Read it as Markdown

    Type to search. Arrow keys move, Enter opens.