# Running several engines

An application can serve several models at once, each on the engine that suits
it: llama.cpp in its own process, and any hosted backend in a **runtime host**.
`praecise_host::Engines` is the one place models are assigned to engines:

- **One engine per model.** Every load claims the model id first; a claim held
  by another engine, or a second host for the same model, is refused with
  `Error::Conflict`. A model is never in memory twice.
- **One memory budget.** Each claim reserves the model's memory from a `Budget`
  shared by every engine: the application's own ledger, or a `FixedBudget`.
- **Confined, not trusted.** A runtime host is a child process the application
  starts by re-running its own executable as the confinement launcher (call
  `praecise_host::sandbox::enter_if_requested()` first thing in `main`). It
  gets new user, network, mount, IPC and PID namespaces, a Landlock ruleset
  (read-only engine environment and model files, one scratch directory, the GPU
  device nodes), a seccomp filter and resource limits. It has no network: no
  interface but an optional loopback with no route out, and no internet socket
  unless loopback is granted. It speaks to the application only over its
  standard streams, one JSON object per line, with request ids so an engine that
  batches sees every outstanding request.
- **Pinned environments.** An `EngineEnv` is a directory holding the engine's
  program, packages and protocol adapter, pinned by the SHA-256 of its whole
  tree (`praecise_host::engine::digest`). The digest is checked before every
  start; an environment that changed is refused, never run.
- **Supervised.** A host that exits fails the requests it was serving and is
  restarted in the background from the same verified environment, while every
  other engine keeps serving. After three failures in a row it is left stopped
  and reported, until the model is released and served again.
- **Accelerator or refusal.** On a machine with an accelerator of any
  vendor, an engine that cannot use it refuses to load rather than falling
  back to the CPU. A host is given the device nodes of every vendor (NVIDIA,
  AMD, the kernel's DRM and compute-accelerator classes, Ascend, Neuron and
  the rest), the vendor runtime directories, and the device-selection
  variables the operator set.

The hosted backends are driven through their offline Python engine APIs (vLLM
`AsyncLLM`, SGLang `Engine`, TensorRT-LLM `LLM`, transformers, OpenVINO GenAI
`LLMPipeline`) by one adapter,
`praecise-host/adapters/praecise_adapter.py`, which `engine::install_adapter`
writes into a provisioned environment before it is pinned. No engine opens an
HTTP server. `praecise_host::generation` maps `GenerationConfig` and
`InferenceResult` onto the protocol, so a hosted engine answers with the same
types, token counts and stop reasons as the linked one; a reply without token
counts is refused rather than metered as free.

A kernel that forbids unprivileged user namespaces (for example
`kernel.apparmor_restrict_unprivileged_userns=1`) refuses the confinement, and
the error names the setting; grant the application's executable `userns` in
its AppArmor profile rather than lowering the setting for the whole machine.
On Ubuntu that is one profile per executable, for example
`/etc/apparmor.d/praecise-job`:

```
abi <abi/4.0>,
include <tunables/global>
profile praecise-job /opt/praecise/bin/praecise-job flags=(unconfined) {
  userns,
}
```

followed by `apparmor_parser -r /etc/apparmor.d/praecise-job`. Where that is
not possible an operator may choose `--confinement landlock-only` explicitly:
the same Landlock ruleset and syscall filter (no internet sockets, no file
outside the rules), without the namespaces. It is never chosen on its own.

## Job engines

Beside the model engines, `praecise_host::job` runs job engines: a fixed
program with typed parameters, input files in, output files and progress out,
run once to completion. A job engine is a pinned environment (the same
SHA-256 tree digest, checked before every run) and one Python adapter,
`praecise-host/adapters/praecise_job.py`, compiled into the binary and placed
read-only beside the run's scratch directory. Every run is confined like a
model host: no network, the environment, model caches and inputs read-only,
one scratch directory writable, the GPU device nodes; it is supervised (time
limit, the whole process tree stopped at the end, the last lines of its error
output on failure); on a machine with a GPU a program refuses rather than run
on the CPU. Parameters are checked against each program's declaration (type
and range) before anything starts, and again by the program.

The programs drive NVIDIA's open simulation systems. They are NVIDIA's own
products and run on CUDA; the ones built on torch also run on AMD ROCm, which
presents as `cuda`:

| Program | System | Out |
|---|---|---|
| `sionna.radio_map` | Sionna RT | radio_map.npz, radio_map.png |
| `warp.example` | Warp | the example's USD stage |
| `newton.example` | Newton | sim.usd |
| `cosmos.generate` | Cosmos 3 (guardrail forced on) | video.mp4 |
| `physicsnemo.darcy_fno` | PhysicsNeMo | fno_darcy.pt and its error on unseen fields |
| `cudaq.ghz`, `cudaq.maxcut` | CUDA-Q | sampled states, the best cut against the optimum |
| `boltz.predict` | Boltz-2 | structures, confidence, affinity |

Each writes result.json. The `praecise-job` binary runs one:

```
praecise-job programs                      # the catalogue, with every parameter's type and range
praecise-job digest /opt/envs/cudaq        # the digest to pin an environment by
praecise-job run --engines engines.json --program cudaq.ghz --param qubits=24 --out out/
```

`engines.json` names each engine's environment and pin:
`{"engines": {"cudaq": {"root": "/opt/envs/cudaq", "digest": "<sha256>",
"read_only": [...], "env": {...}}}}`. Progress is printed as JSON lines and
the last line is the result.
