Task: Migrate the storage consumers onto the shared interface
Table of Contents
This page documents a task in the Give object storage a NATS and HTTP interface with authentication story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Every component that moves bytes calls the storage interface rather than its own copy of it, and the copies go. One HTTP client, one archiver and one set of key builders remain in the tree, the shell's storage verbs are the raw surface rather than five verbs named after their callers, and no consumer reaches storage through a hand-rolled path, key or retry loop.
The buckets stay the consumers' business: each names its own, and nothing in storage needs to learn the names for the migration to work.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Give object storage a NATS and HTTP interface with authentication |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-09-26 |
3. Acceptance
ores.compute.wrapperlinks the storage interface and its own HTTP client and archiver are deleted, including the download path that opens its destination before it reads the status.ores.oreandores.reportingreach storage through the interface, and thereport-databucket name each of them duplicates lives in one place per component rather than in two anonymous namespaces.- The shell exposes the raw surface — a bucket, a key and a local path — and the
consumer verbs (
ore upload,compute publish-package,compute download-input,compute download-output) are thin wrappers over it. - The base URL is plumbed one way, not three, and no service defaults to a port the server does not listen on.
- A grep for a second Beast client, a second libarchive archiver or a hand-built storage path outside the component returns nothing.
4. Plan
(Implementation strategy. Written when work starts; key decisions
are distilled into the parent story's * Decisions at close, but the
plan itself stays — it is the historical record of what we did.)
5. Notes
6. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
7. PRs
| PR | Title |
|---|---|
| #2230 | [storage] Give object storage one bucket, one key protocol and one token |
8. Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|
9. Result
The platform writes one bucket and one key grammar, and the HTTP boundary verifies the tokens the platform issues.
What it delivers
object_keysstates and enforces the protocol – bucketores, keyservice/purpose/id[/name]– on the Object Storage page, with build, parse and the refusal of a segment that could leave the bucket, and key helpers for compute and ore on top.- compute, ore, reporting and the shell write it; the
compute,ore-importsandreport-datanames are gone, the seeded compute package URIs moved with them, and the four consumer roles holdstorage::objects:readandwrite. - The HTTP boundary verifies the platform's RS256 session tokens through IAM's public key, keeping the symmetric authenticator for the login token it mints itself, and it fails closed when no verifier is configured.
- A headers-only api part is generated as an INTERFACE library rather than a source glob CMake refuses.
- The setup SQL is idempotent through the create phase: the entity template drops a policy before creating it, and the hand-written policies carry the same guard.
Superseded by main
Main implemented the bearer plumbing its own way – storage_transfer takes the
caller's bearer and the generated routes relay it – along with the DQ artefact
index fix, the DQ publish replace gate and the storage service variables. Those
commits were dropped in the rebase rather than merged twice.
Verified
compass db recreate completes; component drift is clean; the full build is green;
the recreated database seeds the storage service account and the protocol package
URIs, and live runs published a compute package, uploaded an ORE directory and
completed an ORE import on the new keys.
Left over
- The raw shell surface – a verb taking a bucket, a key and a path – is not built, so the consumer verbs remain the way to reach storage from the shell.
- A second
setup_schema.sqlrun still stops, now on the notify triggers (mq_queues_create.sql:90is the first), so the seed helpers' idempotency is code-reviewed but not yet shown by a clean second run. - The trading and marketdata export handlers are main's own and were not changed here.