Compute Grid Design

┌──────────────────┐    cron job      ┌──────────────────────────────────────┐
│  ores.scheduler  │─────────────────▶│          ores.compute.service        │
│  (reaper job)    │ compute.v1.      │  save_workunit  │ result.submit      │
└──────────────────┘ work.reap        │  (dispatch)     │ (validate+assim.)  │
                                      └────────┬─────────────────┬───────────┘
                                               │ NATS JetStream  │ PostgreSQL
                                      ┌────────▼─────────────────▼───────────┐
                                      │  NATS JetStream   │   PostgreSQL      │
                                      │  stream: COMPUTE  │   ores.compute.*  │
                                      └────────┬──────────────────────────────┘
                                               │ JetStream pull consumer
                                      ┌────────▼─────────────────────────────┐
                                      │           Wrapper (on-node)           │
                                      │  Pulls work · Runs engine · Submits  │
                                      └──────────────────────────────────────┘

1. The DB is the authoritative state store. All lifecycle transitions happen as DB writes first; NATS carries the triggers, not the state.
2. Nodes never connect to PostgreSQL. All DB interaction goes through ores.compute.service.
3. NATS JetStream replaces PGMQ for the work queue. The save_workunit handler publishes workunit assignments synchronously to a durable JetStream stream; wrappers use a pull consumer so work is never double-dispatched.
4. No in-process background loops. The Dispatcher, Validator, and Assimilator are all event-driven and run inline within their triggering handler calls. Only the Reaper needs periodic scheduling, which is delegated to ores.scheduler via a cron job.
5. Multiple instances of ores.compute.service form a NATS queue group; NATS routes each request to exactly one instance automatically.
6. Multi-tenancy: all NATS subjects are prefixed with the environment prefix (e.g. ores.dev.local1.compute.v1.hosts.list).

All state machine transitions except reaping are triggered directly by handler calls — no polling:

All subjects below are relative (the prefix is prepended automatically).

Stream name : COMPUTE
Subject     : compute.v1.work.assignments.*   (* = tenant_id)
Retention   : WorkQueuePolicy (message deleted after ack)
Replicas    : 1 (dev) / 3 (prod)
Max age     : 24h (undelivered assignments expire)

The save_workunit handler publishes one message per result row it creates, containing result_id and workunit_id. The wrapper uses a durable pull consumer with AckWait equal to the expected max engine runtime. If the wrapper crashes before acking, the message re-delivers automatically.

1: Inactive   ──[save_workunit publishes to JS]──▶  2: Unsent
2: Unsent     ──[wrapper calls work.pull]─────────▶  4: InProgress
4: InProgress ──[wrapper calls results.submit]───▶  5: Done
4: InProgress ──[JS AckWait expires]─────────────▶  2: Unsent  (auto re-deliver)
4: InProgress ──[reaper detects stale host]──────▶  2: Unsent  (republish to JS)
5: Done       ──[Validator accepts]──────────────▶  canonical_result_id set on workunit

Create ores.compute.service as a separate CMake project (like ores.scheduler.service).

Structure:

projects/ores.compute.service/
  CMakeLists.txt
  src/
    CMakeLists.txt
    app/
      application.hpp
      application.cpp
      main.cpp
  include/
    ores.compute.service/
      app/
        application.hpp

The service executable wires together:

1. Database context (connection pool, 4 connections).
2. NATS client (connect, set subject prefix).
3. JWT verifier (fetch JWKS from IAM, exponential backoff).
4. Messaging registrar (ores::compute::messaging::registrar::register_handlers).
5. Eventing pipeline (PostgreSQL LISTEN/NOTIFY → NATS publish):
   • ores_compute_hosts_tbl → ores.compute.host_changed
   • ores_compute_results_tbl → ores.compute.result_changed
   • ores_compute_batches_tbl → ores.compute.batch_changed
6. Graceful shutdown: drain NATS, return.

src/CMakeLists.txt dependencies:

target_link_libraries(${exe_target_name}
    PRIVATE
        ores.compute.lib
        ores.service.lib
        ores.eventing.lib
        ores.security.lib
        ores.nats.lib
        ores.database.lib
        ores.utility.lib
        ores.platform.lib)

Register in projects/CMakeLists.txt after ores.compute.

The wrapper binary must authenticate before pulling work. The recommended approach:

• When a host is registered via compute.v1.hosts.save, the handler issues a machine-to-machine JWT (via ores.iam.service) with sub = host_id.
• The wrapper stores this JWT and presents it as a Bearer token in the Authorization header of every NATS request.
• All handlers validate the JWT using the standard jwt_authenticator.

The Dispatcher, Validator, and Assimilator do not need polling loops because they are always triggered by a caller:

• Dispatcher runs inside save_workunit (synchronous, before returning to client).
• Validator runs inside submit_result (synchronous, before returning to wrapper).
• Assimilator runs inside submit_result after the Validator completes.

Only the Reaper needs a timer, and that is provided by ores.scheduler.

PGMQ is not used anywhere in the current codebase. NATS JetStream with WorkQueuePolicy provides equivalent at-least-once delivery semantics via AckWait-based re-delivery, integrates naturally with the existing NATS client library, and avoids a PostgreSQL extension dependency.

pg_cron schedules jobs inside the database process, making them hard to observe, test, and operate. The Reaper's periodic trigger is delegated to ores.scheduler, which already has cron support, NATS integration, and operational visibility.

Multiple instances of ores.compute.service subscribe to the same queue group. NATS routes each wrapper request to exactly one instance automatically — no separate HTTP server pool or proxy is needed.

JetStream handles re-delivery when AckWait expires (wrapper crashed before acking). The Reaper handles a separate edge case: the wrapper acked the message and began running, but then the host went silent without calling results.submit. The Reaper detects this via last_rpc_time staleness and re-queues the result independently of JetStream.