Data storage
Table of Contents
1. Summary
ORE Studio keeps its durable data in two stores, holds disposable copies in a third, and passes work in flight through a stream. The relational database is the system of record: it holds the entities, their versions and their bitemporal history, and it is the only store a caller can query by value. Object storage holds bulk bytes — packages, input and output archives, import bundles and report packages — under a bucket and an opaque key rather than under a schema. Caches and mirrors hold copies that either of the two can rebuild.
The stream is JetStream, the persistence layer inside NATS. The system uses it for events that must survive a service restart, but it holds work in flight rather than the record of what happened.
This page is the entry point to the cluster. It names what each store owns, which interface reaches it, and the order to read the pages in.
2. Detail
2.1. Reading order
Read the store you are about to use, rather than all of them. The store decides the interface, the failure set and the consistency the caller gets, so the choice comes before the design.
- A row you will query, join, version or audit — start at PostgreSQL: Database Architecture and Conventions.
- Bytes with no schema, too large or too shapeless for a row — start at Object Storage.
- A copy held for speed and safe to lose — start at Where should an entity-mirror cache live?.
- An event another service must receive after a restart — start at Message Queue.
2.2. The relational database
PostgreSQL is the system of record. Every entity, its audit columns, its version rows and its bitemporal intervals live here, and the generated SQL, the repositories and the handlers all read and write it.
- PostgreSQL: Database Architecture and Conventions — the store itself: schema layout, roles, session settings, extensions, the key SQL patterns and the bitemporal design.
- Time and Timestamps: Architecture and Conventions — how time is
represented and stored, and the type chain from
db_timestamptostd::chrono::system_clock. - Optimistic Concurrency Versioning — how a concurrent write is detected, and the version sentinel that conflates four write cases.
- Temporal Composite Entity Versioning: Target State — the target for parent-and-child entities, where a child save must bump the parent's version.
- PostgreSQL Row-Level Security — tenant isolation and service-role separation enforced by the database rather than by the caller.
- ores.sql — the component that owns the DDL, the migrations and the role provisioning.
2.3. Object storage
Object storage holds the bytes no table should hold: an application package, a workunit input or output archive, an ORE import bundle and a report package. An object is a bucket, an opaque key and a byte string, so the store never parses what it holds and never learns what a bucket means.
- Object Storage — the target state: one contract over HTTP and NATS, raw use and wrapped use, and a replaceable backend.
- ores.storage — the component as it stands today, and the HTTP client the callers link.
- Compute job lifecycle — the largest consumer, and the bucket and key scheme that compute owns.
2.4. Caches and derived copies
A cache is a copy held for speed, and it is not a store of record. Its placement rule names the component that owns it, and its contents must be rebuildable from the database or from object storage.
- Where should an entity-mirror cache live? — the placement rule for the cached copy of an entity.
2.5. The durable stream, and what it is not
JetStream is NATS's built-in persistence layer. A stream is an append-only, subject-filtered log with consumer cursors, so it holds a message until its consumer acknowledges it — which is what work assignment and event delivery need. The system uses it for events that must survive a service restart.
It is not a store of record. A stream is addressed by subject, not by key, and its retention is a policy the deployment sets rather than a schema the domain declares. The bus, its namespace and its message contract are described in Message Queue and NATS.
2.6. Choosing a store
- A value the system must query, join, version or audit belongs in the database.
- A value the system only moves, whose size or shape no schema describes, belongs in object storage.
- A value either of the two can recompute belongs in a cache, if it belongs anywhere at all.
- A value that must reach another service travels on a stream until that service acknowledges it; the stream retains it by the deployment's policy rather than by the domain's.
3. What this cluster does not cover
- The bus itself: Message Queue, its external substrate NATS, and the entity verbs NATS Entity Protocol Specification.
- How a component's in-memory data model is chosen: Data-Oriented Design in ORE Studio.
- The schema as drawn: PlantUML ER diagram conventions.
- How a stored version becomes a diff: History Diff Architecture.
4. See also
- Component Architecture — where a store's client sits in the api, core and service split.
- Entity Lifecycle — the layer order a stored entity passes through.
- Component Clean Standard — the rules a storage component passes to be clean.