Compute job lifecycle

The grid uses the BOINC vocabulary: a batch holds workunits, each workunit is executed by one result per redundancy copy, and a host is a worker node. apps, app versions, and platforms describe what runs where.

The batch statuses and the result server_state transitions are shown in compute_job_lifecycle_states.puml:

Source: compute_job_lifecycle_states.puml. Regenerate with plantuml -tpng doc/knowledge/architecture/compute_job_lifecycle_states.puml.

Two points are worth stating explicitly because they are easy to misread:

• The workunit has no status column. Its state is derivable: it has no canonical_result_id yet, or it has one (the job is done).
• The dispatcher creates the result row as Unsent. The claim moves it to InProgress and binds the claiming host. The submit message moves it to Done. If the host stops extending its lease, the lease reaper returns the entry to the queue, and the next claim runs the job again. A job that uses its three attempts ends as Done with outcome NoReply.

The interaction diagram below is the map for everything that follows:

Source: compute_job_lifecycle_sequence.puml. Regenerate with plantuml -tpng doc/knowledge/architecture/compute_job_lifecycle_sequence.puml.

The lifecycle has six stages:

1. Register the engine — upload the ORE package and register the app.
2. Prepare an input bundle — the job's working-directory snapshot.
3. Create the batch — with the generated batches add command.
4. Dispatch — with compute dispatch-batch; the service stores workunits and queues one result per job.
5. Execute on a node — the wrapper downloads, verifies, runs, and uploads the output.
6. Validate and close — the service accepts results, marks the workunit, and closes the batch; the operator downloads outputs.

The engine package is the self-contained tarball a wrapper downloads and runs. It holds the ore executable, every non-glibc shared library it needs (under lib/, RPATH-relative), and the minimal manifest.json the wrapper reads. ORE Studio vendors one package per engine build under external/ore/packages/; the seed registers the ORE 1.8.17.0 build, ore-1.8.17.0-x64-linux.tar.gz, and the older ore-1.8.16.0-4-g3b62ba248-x64-linux.tar.gz stays vendored beside it:

ore-1.8.17.0-x64-linux.tar.gz
├── manifest.json          {"executable": "ore-1.8.17.0-x64-linux",
│                           "args": ["Input/ore.xml"]}
├── ore-1.8.17.0-x64-linux
└── lib/                   every shared library the engine needs

The manifest tells the wrapper which executable to spawn and with which arguments. For ORE the argument is the job's input file Input/ore.xml, resolved relative to the job's working directory — which is exactly how a packaged Academy example is laid out. The wrapper runs the engine with the job directory as its working directory, so Input/, Output/, and log.txt inside ore.xml all resolve inside that directory. See Compute Engine for the engine itself and external/ore/packages/README.md for how packages are built and self-tested.

Upload the vendored package with compute publish-package. The verb uploads the tarball to storage under the app's package key, computes and verifies its SHA256 server-side, and registers (or merges into) the app, app_version, and app_version_platform rows:

connect $ORES_NATS_URL
login super_admin@localhost Secure-Password-123
compute publish-package ORE 1.8.17.0 x64-linux \
    --wrapper-version 1.0.0 \
    --file external/ore/packages/ore-1.8.17.0-x64-linux.tar.gz
logout
exit

publish-package merges rather than duplicates: the app row is keyed by name, the app-version row by (name, engine_version, wrapper_version), and the platform rows of a version are replaced wholesale (previous platforms are preserved and the published one is updated). Passing --wrapper-version 1.0.0 lines the upload up with the row that projects/ores.sql/populate/compute/compute_ore_app_seed.sql seeds — run that seed, or run compute_populate.sql, to register the same app plus the x64-linux platform row that compute_platforms_seed.sql provides. The seed alone registers only the rows; the package bytes still need the publish-package upload above.

Storage keys. All blobs live in the compute bucket of ores.storage (/api/v1/storage/compute/ over HTTP). The seeded ORE package is registered at:

Every job of a batch runs against the same input bundle: a tarball whose root holds the job's input directory, exactly as the engine expects it. ORE Studio vendors the sample job external/ore/packages/TA002_IR_Swap.tar.gz, built from the Academy example TA002_IR_Swap (a 20-year EUR fixed-vs-float interest-rate swap). Its layout is the working-directory snapshot the engine will see after extraction:

TA002_IR_Swap.tar.gz
└── Input/
    ├── ore.xml              the run's main config (inputs, Output/, log.txt)
    ├── market.txt, fixings.txt
    ├── todaysmarket.xml, pricingengine.xml, curveconfig.xml,
    │   conventions.xml, irswap.xml

The sample's ExpectedOutput/ (the curves.csv, flows.csv, npv.csv the engine should produce) lives in the Academy example and is the reference for the package self-test, not part of the job bundle. To build a bundle for another job, tar the input directory the same way — the archive is uploaded verbatim, and the manifest argument Input/ore.xml is what pins the engine to that layout.

The generated batches add creates a batch row with status open. The batch is only a container plus an operator-chosen external_ref; nothing runs yet:

connect $ORES_NATS_URL
login super_admin@localhost Secure-Password-123
batches add ir-swap-2026-09-06 open system.new_record "first run"
logout
exit

An --smoke batch is bounded to 10–20 jobs and is what the smoke test uses. See How do I add a compute batch from the shell?.

compute dispatch-batch takes the batch reference, the job count, the app-version id, and the input tarball:

app_versions list                  # note the id of the ORE 1.8.17.0 row
compute dispatch-batch ir-swap-2026-09-06 15 \
    <app_version_id> external/ore/packages/TA002_IR_Swap.tar.gz

Only an open batch can be dispatched, and the app version must exist (the command resolves it before uploading anything). Dispatch then has three parts:

1. Upload the input bundle — the tarball goes verbatim to input/{batch_id}.tar.gz; every workunit of the batch references that one shared bundle.
2. Store the workunits — one workunit row per job, each with priority 1, target_redundancy 1, and input_uri pointing at the shared bundle. The batch status flips to dispatched only after every workunit is saved; a save failure partway leaves the batch open so dispatch can be retried.
3. Dispatch — each stored workunit raises a domain event that the service's eventing listener feeds into the workunit_dispatcher. For every workunit short of its redundancy target the dispatcher creates a result row (server_state Unsent), queues it in the work ledger with the platforms that can run it, and rings the compute.v1.ops.work_available notice so idle nodes ask at once. A dispatch that would pass the tenant's limit of waiting jobs is refused.

The ledger entry holds ids only. When a node claims the job, the service builds the assignment from the workunit: package_uri and package_sha256 come from the app version's published platform row (hence the warning if a version has no package: the workunit is stored, no entry is ever claimed, and the job sits Unsent forever). output_uri is pre-assigned as output/{result_id}.tar.gz. The storage token is minted at the claim, so it is fresh however long the job waited.

The dispatcher converges on the redundancy target: a workunit whose result count is already at target gets no new result, and a partially failed dispatch is topped up by the next workunit event instead of stranding the workunit. See How do I dispatch a compute batch from the shell? for the recipe.

A wrapper node is an ores.compute.wrapper process pointed at the environment (--host-id is mandatory; the node also needs its tenant, work_dir, heartbeat interval, and the storage HTTP base URL). On startup it starts a claim loop. The loop sends compute.v1.ops.claim_work with the host id and its own platform triplet, and the service answers with one job or with no work:

request:  host_id, platform
reply:    result_id, workunit_id, package_uri, package_sha256,
          input_uri, output_uri, storage_token, attempt

A node asks only when it is free, and asks again at once after a job. When no work waits, it waits for the work_available notice, or for a long timeout measured in minutes that covers a lost notice, and sends nothing until one of them comes (Work is pushed to the grid, and an idle grid costs one claim per wrapper per timeout). The service chooses the job: the platform decides which entries a node can run, the tenant with the fewest running jobs goes first, and then priority and age. The claim holds the job under a lease of five minutes. The heartbeat extends the lease, and a node that dies mid-run loses it. The lease reaper then returns the job to the queue, or ends it as no reply after three attempts.

Every heartbeat_interval (30 seconds by default) the node publishes compute.v1.ops.heartbeat with its host id. The first heartbeat auto-registers the host row and gives it a deterministic adjective-animal display name carrying an eight-hex-digit discriminator from the same id (e.g. clever-echidna-4f2a9c31), so two nodes are unlikely to share a name by drawing the same words. A heartbeat from a known host writes nothing: liveness is a series, and the grid judges a host online from the recency of its newest node sample. When telemetry is enabled the node additionally publishes node samples on compute.v1.ops.node_sample.

On each claimed job the wrapper runs process_assignment:

1. Ensure the package — work_dir/packages/{app_version_id}/ holds the extracted engine. If the manifest.json is missing, the node downloads package_uri, verifies its SHA256 against package_sha256 (a mismatch aborts the job — a corrupted or tampered package is never run), and extracts the archive.
2. Prepare the job directory — work_dir/jobs/{result_id}/ is created; the input archive is downloaded from input_uri and extracted there. The engine will run with this directory as its working directory.
3. Run the engine — the executable named in the manifest is spawned (Boost.Process; no shell involved) with the manifest arguments and the job directory as start_dir. For ORE that is Input/ore.xml relative to the job directory; ORE writes its run log to Output/log.txt there. The engine's own stdout/stderr go to engine.log in the job directory, with the exact command line logged for reproduction.
4. Finish — on exit code 0 the wrapper archives the job's Output/ directory and uploads it to output_uri; on any other exit code there is no archive, and the error message carries the tail of Output/log.txt. The engine logs are published to the telemetry log-entry subject for the record.
5. Submit — the node sends a request/reply submit over compute.v1.ops.submit_result, the trusted machine channel (wrapper nodes hold no user JWT, so the user-session-gated CRUD flow cannot accept their submissions):

subject: compute.v1.ops.submit_result
payload: result_id, host_id, output_uri,
         outcome (1=Success, 3=ClientError), error_message

The submit handler (in ores.compute.core, queued as ores.compute.service so one replica validates) is the validator and assimilator of the grid:

1. Record the result — the result row is marked Done with the outcome, the submitting host bound as host_id, the output URI, and a received timestamp. A submit from a host that does not hold the claim, or for a result that is already done or retired, changes nothing.
2. Accept on the workunit — once the workunit has target_redundancy (1 in shell dispatch) results in state Done, the validator records the last-submitted result's id as the workunit's canonical_result_id. Acceptance does not filter by outcome: with redundancy 1 a failed job (ClientError) still satisfies the target, so the grid does not re-dispatch it. Outcome checking is the operator's, through compute grid-stats or the result list — the smoke-test script asserts every result has outcome Success and fails otherwise.
3. Close the batch — when every workunit of the batch has a canonical result, the batch status moves to closed (even if some of those canonical results failed).

The operator watches the drain and collects outputs from the shell:

compute grid-stats --watch ir-swap-2026-09-06      # poll until the batch drains
workunits list                                     # canonical_result_id per job
results list                                       # result rows: state, outcome, host
compute download-output <result_id> <dest_dir>     # fetch output/{result_id}.tar.gz

The downloaded archive is the Output/ directory the engine wrote — for the TA002 sample, the curves.csv, flows.csv, and npv.csv that the Academy example's ExpectedOutput/ predicts. See How do I view compute grid stats from the shell? and How do I download a compute result output from the shell?.

The smoke-test library script drives the entire lifecycle in one session: it adds a 15-job --smoke batch, dispatches it to the target app version with the sample input, watches grid-stats until the batch drains, and asserts that every online host was exercised and every result has outcome Success. See How do I smoke test the compute grid from the shell?.