Data-Oriented Design in ORE Studio
Table of Contents
1. Summary
Data-oriented design (DOD) organises data by how code accesses it, not by the conceptual hierarchy of the domain. In ORE Studio it governs two regimes: numerical kernels (pricing, simulation, analytics) where layout decides cache and vectorisation performance, and persisted domain data (entities, reference data, trades) where the value-semantics and shape of the C++ types decide cost. A third regime, the wire format, applies the same axioms to NATS messages and JSON. The compass-code-review-data-model skill reviews a component's data model against the criteria below; this page is the canonical statement of those criteria.
2. Detail
2.1. The principle and the axioms
The first statement is the founding principle of DOD. The five that follow are the operational axioms derived from it. Only the axioms produce review checks. The principle explains why the axioms hold; it is not itself a check, so never score it.
- Data is memory — the principle. Code exists only to transform data from one representation to another. The representation decides the cost.
- Access patterns drive layout. Organise data by how and when code reads and writes it together, not by real-world taxonomy.
- Structure of Arrays (SoA) over Array of Structures (AoS). Prefer parallel contiguous arrays per attribute over arrays of fat heap-allocated objects.
- Separate hot and cold data. Isolate state read or written on high-frequency paths from metadata read rarely or once.
- Handles over pointers. Use keys, indices, and offsets instead of per-object heap pointers and object trees.
- Batch over per-object. Write functions that process contiguous collections in bulk, not one call per object.
2.2. There is no entity
The rule is section 5.4 of Richard Fabian's Data-Oriented Design (2013), There is no Entity (dodmain/node5.html). The section names the step of turning a class into "containers of smaller objects, an approach often called composition" as a transitional state, and puts the destination beyond it: "A component based object is nothing more than the sum of its parts", and its definition is "an inventory with some construction arguments".
The reading this project takes is the one the trading model wrote down: no object graph is modelled; rows are atomised and keyed; relationships are foreign keys and joins; there is no parent that owns children, and no aggregate versions as a whole. See dfe15809, the trading data-oriented model.
Three consequences bind this project, and all are reviewable.
- The components are separate types. No aggregate type represents an entity, whether its parts are nested members or a read-time aggregate. Each component has its own type, its own key, and its own codec.
- There is no entity object, so no entity object travels. A message carries one typed array per component, joined by key. One component's fields never appear as nested objects inside another.
- The unified view is assembled for display. A screen that needs one row from several components reads the arrays for one page and zips them. The server keeps filtering, ordering, paging and the total, so the arrays stay aligned. A screen that spans several entities is a read model, never a client join; see NATS entity protocol specification.
This is the target for every component, not for one. Codegen does not
reach it yet: it emits one aggregate type per entity and projects that
type directly, so a grouped entity's members appear nested on the wire,
as the trading instrument fixtures do with identity. The atomised shape
is a codegen change, and until it lands an aggregate is a known defect
against this rule rather than a licence for it. The atomised shape also
relieves the translation-unit pressure the C1202 section records, because
no unit reflects one wide entity type.
The target shape, the split rule, the part naming and the read are in The atomised entity: a central row and its parts.
2.3. Where each axiom applies
ORE Studio is a database-backed application with Qt clients and a quantitative engine on QuantLib. DOD is not one rule for the whole codebase. A review that demands SoA for a CRUD entity, or accepts a pointer graph inside a simulation kernel, has misread the regime.
2.3.1. Regime 1 — numerical kernels
The hot paths: valuation, simulation, curve bootstrapping, analytics, compute workunits. Axioms 2-6 apply in full. Typical evidence: a loop over trades, paths, or steps that reads a few fields of each object.
2.3.2. Regime 2 — persisted domain data
Entities and their C++ value types: reference data, trading instruments, market data, synthetic entities. The database owns the storage layout; the C++ types are the in-memory shape, and the wire is their flat projection. DOD here means value semantics, identity by key, and a clean split of core state from cold annexes — not cache-line gymnastics.
2.3.3. Regime 3 — wire and messaging
NATS messages, JSON, CSV, and table I/O. The axioms apply as: flat value-shaped messages, versioned schemas, no polymorphic payload fields, and batched messages for high-volume feeds.
A message carries no std::variant field and no std::string field holding
an encoded value. A heterogeneous result set travels as one typed array per
concrete type, joined by key, so each element's type is static from the array
it sits in. The tag belongs to the container, not to every element. See
Polymorphic payloads over NATS
for the deprecated pattern this replaces.
The wire shape is a codegen projection of the Regime 2 entity, not a second artefact: one entity yields one JSON codec and one table codec. The projection is flat. A grouped C++ type is projected as flat value fields, per There is no entity above; its members are a layout decision for memory and storage, not a wire contract. Review each entity once — its org model, its generated C++ type, and its projections together — and do not walk the message shape again separately. Hand-written message types are the exception; they get their own Regime 3 walk.
2.3.4. The boundary of the regimes
The three regimes cover table-shaped data and its projections. They
are a labelled cover of this codebase, not a partition of every
possible data shape. One category sits outside them: document-shaped
payloads — an FpML trade document, an ORE XML input, an org-mode
file parsed into a heading tree. Such documents enter at the
boundary of the stack, where a mapping layer converts them to and
from the core data-model representation (for example, an FpML trade
maps to trade and its instrument entities). The document tree is
transient; the entity representation, designed for performance, is
the review target.
For a document-shaped type, mark the table-shaped-children and
flat-value-message checks as not applicable and say why. In-tree
today the category exists only as hand-written core types in
ores.orgmode (the heading/document structure); no entity model
instantiates it yet. Revisit this paragraph if an entity model ever
persists document trees.
A document is not a classification of data on its own. It is an artefact at the boundary, and the data it becomes is classified separately, per the next section.
2.4. Classification precedes the pattern
The three regimes say where an axiom applies. They do not say what a set of data is, and they are not a substitute for asking. A modeller classifies the data first, then chooses the pattern for that classification: Data classifications: entity, list, family, bundle, package and artefact. It is a closed set of six: an entity, a list of one entity kind, a family written in one transaction, a bundle of related families that one owner applies in one unit of work, a package that carries bundles between systems as bytes, and a boundary artefact. The same page names the anti-type — a hand-written union of unrelated entities presented as one struct — and the test that refuses it.
The rule that decides a set of different types is one question: do the families share an invariant that must hold at one instant, or one operation whose intent is all-or-nothing? With no relatedness, split the set into per-family or per-entity messages; if the bytes must still move, they are a package. With relatedness and one owner, use a bundle. With relatedness and several owners, no bundle spans them: use a package carrying one bundle per owner. A file's contents are not an invariant; a file is an artefact.
The form a modeller fills in, and the review artefact, is The data classification checklist.
Classification precedes the pattern, and it also outranks a compile error. A type that is correctly classified and still does not compile has a technical problem; denormalise for the compiler if the shape needs it, and keep the classification. A compile error never reclassifies the data.
2.5. Decomposition and the MSVC C1202 ceiling
The generated codecs have replaced compile-time reflection, so this
limit no longer applies to them. It is recorded because the decomposition
it forced is still in the models. A reflected struct had a hard shape
limit that nothing in the type system expressed, and crossing it failed
the build on MSVC with C1202.
MSVC capped the recursive type-dependency graph that reflection built. The count that mattered was the number of field names in one reflected literal, and that literal depended on how the type was written:
- A plain struct gets a literal of its own members. A nested struct member adds one name, not its members' names. Generated entities with as many as 59 plain members compile on MSVC; on 2026-10-03, 220 of the 287 reflected structs the models generate had more than nine members.
- Flattening a member adds the flattened struct's names to the parent's literal, recursively. Flatten is a JSON-shape decision, never a decomposition fix.
- Writing a
std::variantwith tags builds one literal over every field name reachable from the alternative being written. This is the shape that has failed: the 502-name trade detail wrapper in sprint 19, and the bond export carrierbond_instrument_dataon main in October 2026.
The limit reaches C1202 by two paths. One is the width of a single
literal, which is why moving a type between headers does not help and no
compiler flag raises it on MSVC. The other is accumulation across a
unit: a unit that reflects many individually safe literals can still
fail. Measured 2026-10-13, the 24-member tenor_basis_swap_convention
literal built in a unit of twelve repository headers and failed inside
the configuration document handler, which reaches 58 document element
types through 103 repository headers.
The rule: keep the names reachable from one tagged write small, by
writing the parts of a large carrier separately rather than as one
tagged aggregate. Splitting a wide entity into field_group members
lowers the parent literal, so it is a remedy for the per-literal path. It
is not a wire decision: the JSON stays flat per There is no entity
above, and a change that nests it has changed a contract to fix a
compile error. No direct-member count is the C1202 limit; the
nine-member figure this page once gave was a margin, not a measured
ceiling.
The accumulation path gives a second rule for document models. A model that aggregates several entity families hands its unit a literal width equal to the sum of them, so name one bundle per document model and split a document that covers two. A component's per-entity protocols do not help here, because a document protocol includes the domain headers directly and re-aggregates what the per-entity split separated.
That rule keeps a document message that a real invariant sanctions. It does not sanction a message that carries families the document merely lists. Where there is no cross-type invariant, the file is an artefact and each family has its own message, per Data classifications.
2.6. The data model of a component
The artefacts a review reads, per component under projects/ores.<name>:
modeling/*.org— the codegen entity, junction, and field-group models (where the component is codegen-driven). Thelookup_entitymetatype is reserved in codegen but has no authored instances; do not hunt for lookup models.- The domain C++ types they generate, or hand-written domain headers
in the
corefacet. This generated representation is the core data model, designed for performance; the review target. - The SQL schema the entity maps to: the
CREATE TABLEstatements generated underprojects/ores.sql/create/<component>/. - Repository and service signatures: how code fetches collections and looks up records.
- Kernel entry points and their inner loops, for Regime 1 components.
2.7. Review criteria
2.7.1. Regime 1 checklist
| Check | Description |
|---|---|
| SoA on the hot path | Fields a kernel iterates sit in parallel contiguous arrays, not in an array of fat objects. |
| No hot-loop allocation | Buffers are reused; no allocate/free or lock inside the loop. |
| No per-object dispatch | The hot path calls batch free functions over spans; no virtual or per-object indirection. |
| Hot/cold kernel inputs | Metadata read once per run is not interleaved with per-step state. |
| Predictable access | Inner loops are stride-1 over contiguous memory so the compiler can vectorise. |
| Plain inputs | Kernels take plain structs and free functions (e.g. a log-normal short-rate SDE), not stateful class hierarchies. |
2.7.2. Regime 2 checklist
Regime 2 review walks two surfaces, and each check below states which one it applies to:
- The decision surface: the
modeling/*.orgmodel choices — key style, field-group split, junction versus blob, batched access. These choices can fail; codegen does not rescue them. Review them per entity model. - The invariant surface: on codegen entities, value semantics,
pointer-freedom, and generated codecs are generator invariants.
Codegen cannot emit a
shared_ptror a hand-rolled codec. Walking a guarantee per entity rubber-stamps it. Spot-check each invariant once per component on the generated output, or rely on the codegen drift check (the compass-codegen-fix-drift skill owns drift). Hand-written domain types in thecorefacet have no generator; walk both tables in full on them.
Decision checks — walk per entity model:
| Check | Description |
|---|---|
| Identity by key | Relationships are keys and foreign keys, not pointers or owning-parent links. |
| Table-shaped children | Child records are rows in their own table (junction/FK), not nested object trees. |
| Core/cold split | field_group and table design separate core fields from cold annexes (change-reason cache, history, versioning, audit). The split is for memory and storage; the wire stays flat. |
| Batched collection access | Repositories and services return collections, not row-at-a-time callbacks. |
| Explicit variation | A group of related entity kinds (for example, instruments) states its variation explicitly — per-type tables or a closed variant — not open inheritance webs. |
| One component per row | Every column of a row holds for every row. Three shapes break this, and each is a part of its own, keyed by the owner, with at most one row per owner: a has_* flag followed by the columns it governs (the part's presence replaces the flag); a discriminator followed by columns that only one of its values uses (one part per value); two legs written as column sets, each leg with its own terms (one part per leg). Two values of one term, such as a threshold each way of one agreement, are two fields of one component and not two legs. A width is never the argument for a split, and a type with none of the three shapes stays whole, however wide. |
| Exact money at rest | A money quantity — rate, spread, notional, amount, price, strike, fee, premium, margin, face value — is exact decimal in both the domain type and the column, and the two agree. A continuous quantity — volatility, correlation, weight, probability, recovery rate, tranche fraction, day-count fraction, unit quantity — stays binary float. |
| One-way numeric conversion | A double never enters the persisted domain. Conversion runs from decimal to double, at the numerical-kernel seam, and only there. A value that arrives as text is read with a decimal parse, not through a binary float. |
Invariant checks — spot-check once per component:
| Check | Description |
|---|---|
| Value semantics | Generated domain types are plain values with a key; no owning-pointer members. |
| No shared-ownership webs | No shared_ptr inside domain types; ownership sits at component boundaries. |
| Codec drift | JSON/CSV/table codecs come from codegen unmodified; regenerate rather than hand-patch. |
2.7.3. Regime 3 checklist
| Check | Description |
|---|---|
| Flat value messages | Messages are value-shaped, not object graphs over the wire. A composite C++ type is projected flat; its members never appear as nested JSON objects. |
| No polymorphic payload fields | No std::variant field and no encoded std::string field; one typed array per concrete type, joined by key. |
| Feed batching | High-volume feeds batch items per message instead of one message per item. |
2.8. Report conventions
A component review produces, per component:
- A regime classification with the reasons (what is hot, what is cold, where the boundaries are).
- A findings table: severity (Critical / Important / Minor), effort (Small / Medium / Large), and the criterion each finding maps to.
- A per-regime score of 1-5 where the regime applies.
- An overall data-model verdict and a debt level of Low / Medium / High.
Scores are anchored, not arithmetic. Pick the row the review most resembles:
| Score | What that looks like | Debt |
|---|---|---|
| 5 | All applicable checks pass; only evidence-free opportunities remain. | Low |
| 4 | Decisions pass with Minor findings; invariants hold. | Low |
| 3 | One Important finding, localised to one model or one kernel. | Medium |
| 2 | Several Important findings, or one Critical finding in a main flow. | Medium |
| 1 | Critical findings on identity, core/cold split, or a hot loop; systemic. | High |
The debt bands follow the score: 4-5 is Low, 3 is Medium, 1-2 is High. A regime that does not apply to the component is unscored, not scored 0. Two reviews of the same component should land within one point of each other; if they do not, the classification or the evidence was unclear, not the scale.
Hot-path claims need evidence: the loop structure, a profile, or a benchmark. Without evidence, record a layout risk as an opportunity, not a defect. A finding that demands a transformation outside the component's regime is a review error, not a finding.
3. See also
- Richard Fabian,
Data-Oriented Design(2013) — the book this page draws its axioms and its section 5.4 rule from. https://www.dataorienteddesign.com/dodmain/ - dfe15809 — the trading data-oriented model, where the project's reading of section 5.4 is written out.
- Code Review Checklist — the sibling criteria set for general component and PR review.
- compass-code-review-component — the broad whole-component audit this lens complements.
- The atomised entity: a central row and its parts — the shape the Entity classification selects.
- Data classifications: entity, list, family, bundle, package and artefact — the closed set a modeller picks from before choosing a pattern, the relatedness test, and the anti-type.
- The data classification checklist — the numbered form a modeller fills in.
- Transporting a package — how a set of several owners' bundles arrives at one instant, atomic per bundle.
- Codegen patterns: choosing a model shape — the pattern library a classification selects from.
- Knowledge — the index this page belongs to.