Task: Migrate the storage consumers onto the shared interface

Table of Contents

This page documents a task in the Give object storage a NATS and HTTP interface with authentication story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

Every component that moves bytes calls the storage interface rather than its own copy of it, and the copies go. One HTTP client, one archiver and one set of key builders remain in the tree, the shell's storage verbs are the raw surface rather than five verbs named after their callers, and no consumer reaches storage through a hand-rolled path, key or retry loop.

The buckets stay the consumers' business: each names its own, and nothing in storage needs to learn the names for the migration to work.

2. Status

Field Value
State DONE
Parent story Give object storage a NATS and HTTP interface with authentication
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-09-26

3. Acceptance

  • ores.compute.wrapper links the storage interface and its own HTTP client and archiver are deleted, including the download path that opens its destination before it reads the status.
  • ores.ore and ores.reporting reach storage through the interface, and the report-data bucket name each of them duplicates lives in one place per component rather than in two anonymous namespaces.
  • The shell exposes the raw surface — a bucket, a key and a local path — and the consumer verbs (ore upload, compute publish-package, compute download-input, compute download-output) are thin wrappers over it.
  • The base URL is plumbed one way, not three, and no service defaults to a port the server does not listen on.
  • A grep for a second Beast client, a second libarchive archiver or a hand-built storage path outside the component returns nothing.

4. Plan

(Implementation strategy. Written when work starts; key decisions are distilled into the parent story's * Decisions at close, but the plan itself stays — it is the historical record of what we did.)

5. Notes

6. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

7. PRs

PR Title
#2230 [storage] Give object storage one bucket, one key protocol and one token

8. Review

Comment summary File Decision Notes
       

9. Result

The platform writes one bucket and one key grammar, and the HTTP boundary verifies the tokens the platform issues.

What it delivers

  • object_keys states and enforces the protocol – bucket ores, key service/purpose/id[/name] – on the Object Storage page, with build, parse and the refusal of a segment that could leave the bucket, and key helpers for compute and ore on top.
  • compute, ore, reporting and the shell write it; the compute, ore-imports and report-data names are gone, the seeded compute package URIs moved with them, and the four consumer roles hold storage::objects:read and write.
  • The HTTP boundary verifies the platform's RS256 session tokens through IAM's public key, keeping the symmetric authenticator for the login token it mints itself, and it fails closed when no verifier is configured.
  • A headers-only api part is generated as an INTERFACE library rather than a source glob CMake refuses.
  • The setup SQL is idempotent through the create phase: the entity template drops a policy before creating it, and the hand-written policies carry the same guard.

Superseded by main

Main implemented the bearer plumbing its own way – storage_transfer takes the caller's bearer and the generated routes relay it – along with the DQ artefact index fix, the DQ publish replace gate and the storage service variables. Those commits were dropped in the rebase rather than merged twice.

Verified

compass db recreate completes; component drift is clean; the full build is green; the recreated database seeds the storage service account and the protocol package URIs, and live runs published a compute package, uploaded an ORE directory and completed an ORE import on the new keys.

Left over

  • The raw shell surface – a verb taking a bucket, a key and a path – is not built, so the consumer verbs remain the way to reach storage from the shell.
  • A second setup_schema.sql run still stops, now on the notify triggers (mq_queues_create.sql:90 is the first), so the seed helpers' idempotency is code-reviewed but not yet shown by a clean second run.
  • The trading and marketdata export handlers are main's own and were not changed here.

Emacs 29.3 (Org mode 9.6.15)