The data path

The data path is how a service reaches the database and how ticks reach the store and the parties, designed to carry one hundred thousand feeds on one machine that runs ten environments. A service holds a few long-lived connections, a transaction carries its own tenant context, repositories work inside an open transaction, statements are typed and prepared once, and every store error says whether a retry can succeed. Handlers run on a bounded pool. Ticks are stored once for each source, written in bulk by lanes, and kept in a bounded, compressed window. The environments share the instance through a pooler, the compute grid is woken rather than polled, and every queue and session reports its own numbers.

This page orders the cluster. Read it top to bottom when changing how a service talks to the store; read one page when changing one piece.

Every page of the cluster states the target, in the present tense: what the data path is once it is built. What is built so far is tracked by the tasks of Carry one hundred thousand feeds on the data path.

The order follows dependency: each page needs the one before it.

1. The data path serves a control plane and a data plane — the workload. It sets the numbers every later page is sized for, and it splits the path in two, which is why the control plane and the data plane read differently from here on.
2. A service holds a few long-lived connections and never connects per query — the connections exist before any query does, and their count is the first budget.
3. A transaction sets its own tenant context, so any connection serves any tenant — what makes those connections interchangeable, so a pool and a pooler can share them.
4. A repository works inside an open transaction and never holds credentials — the only way code reaches a connection, built on the two pages before it.
5. A hand-written statement is declared once, typed both ways and prepared once — what runs inside that transaction when no generated repository fits.
6. A store error says whether a retry can succeed — what comes back out of it, and what the caller does about it.
7. Delivery threads enqueue and a bounded pool runs the handlers — who runs the handlers, and why their number equals the connections' of page 2. It follows page 6 only because it closes the control plane.
8. A source tick is stored once and entitled when it is read — the data plane begins: what the store holds, and why a party's entitlement moves to the read.
9. Ticks reach the store through sharded binary COPY in bounded batches — how the points get there, given that they hold no tenant.
10. The tick store is bounded — how much of them is kept, given that they carry no row-level security and so compress.
11. The environments share the database through a transaction pooler — how ten environments share the instance, which needs the stateless connections of page 3 and the prepared statements of page 5.
12. Work is pushed to the grid, and an idle grid costs one claim per wrapper per timeout — the compute grid, which applies the same rule to its claims: no query runs until there is work.
13. Every queue and every session reports its own numbers — how every page above reports what it costs, so a change shows as a number.

• The tick ingest pipeline: freshness first — the lane, the ring buffer, the age gate and the overload states of the tick ingest in full. Read it when working on a lane.
• The tick ingest pipeline design record — the measurements, the practice and the options set aside behind the pipeline.
• PostgreSQL: Database Architecture and Conventions — the conventions of the store itself: schema, roles, bitemporal design. Read it when writing SQL.

• The bus, its subjects and its message contract: Message Queue.
• The market data model, from vendor to party: Market Data Architecture.
• Which store a value belongs in: Data storage.

• Knowledge index — where this structure note is listed with the others.