Data storage

Table of Contents

1. Summary

ORE Studio keeps its durable data in two stores, holds disposable copies in a third, and passes work in flight through a stream. The relational database is the system of record: it holds the entities, their versions and their bitemporal history, and it is the only store a caller can query by value. Object storage holds bulk bytes — packages, input and output archives, import bundles and report packages — under a bucket and an opaque key rather than under a schema. Caches and mirrors hold copies that either of the two can rebuild.

The stream is JetStream, the persistence layer inside NATS. The system uses it for events that must survive a service restart, but it holds work in flight rather than the record of what happened.

This page is the entry point to the cluster. It names what each store owns, which interface reaches it, and the order to read the pages in.

2. Detail

2.1. Reading order

Read the store you are about to use, rather than all of them. The store decides the interface, the failure set and the consistency the caller gets, so the choice comes before the design.

2.2. The relational database

PostgreSQL is the system of record. Every entity, its audit columns, its version rows and its bitemporal intervals live here, and the generated SQL, the repositories and the handlers all read and write it.

2.3. Object storage

Object storage holds the bytes no table should hold: an application package, a workunit input or output archive, an ORE import bundle and a report package. An object is a bucket, an opaque key and a byte string, so the store never parses what it holds and never learns what a bucket means.

  • Object Storage — the target state: one contract over HTTP and NATS, raw use and wrapped use, and a replaceable backend.
  • ores.storage — the component as it stands today, and the HTTP client the callers link.
  • Compute job lifecycle — the largest consumer, and the bucket and key scheme that compute owns.

2.4. Caches and derived copies

A cache is a copy held for speed, and it is not a store of record. Its placement rule names the component that owns it, and its contents must be rebuildable from the database or from object storage.

2.5. The durable stream, and what it is not

JetStream is NATS's built-in persistence layer. A stream is an append-only, subject-filtered log with consumer cursors, so it holds a message until its consumer acknowledges it — which is what work assignment and event delivery need. The system uses it for events that must survive a service restart.

It is not a store of record. A stream is addressed by subject, not by key, and its retention is a policy the deployment sets rather than a schema the domain declares. The bus, its namespace and its message contract are described in Message Queue and NATS.

2.6. Choosing a store

  • A value the system must query, join, version or audit belongs in the database.
  • A value the system only moves, whose size or shape no schema describes, belongs in object storage.
  • A value either of the two can recompute belongs in a cache, if it belongs anywhere at all.
  • A value that must reach another service travels on a stream until that service acknowledges it; the stream retains it by the deployment's policy rather than by the domain's.

3. What this cluster does not cover

4. See also

Emacs 29.3 (Org mode 9.6.15)