Data-Oriented Design in ORE Studio

Table of Contents

1. Summary

Data-oriented design (DOD) organises data by how code accesses it, not by the conceptual hierarchy of the domain. In ORE Studio it governs two regimes: numerical kernels (pricing, simulation, analytics) where layout decides cache and vectorisation performance, and persisted domain data (entities, reference data, trades) where the value-semantics and shape of the C++ types decide cost. A third regime, the wire format, applies the same axioms to NATS messages and JSON. The compass-code-review-data-model skill reviews a component's data model against the criteria below; this page is the canonical statement of those criteria.

2. Detail

2.1. The principle and the axioms

The first statement is the founding principle of DOD. The five that follow are the operational axioms derived from it. Only the axioms produce review checks. The principle explains why the axioms hold; it is not itself a check, so never score it.

  1. Data is memory — the principle. Code exists only to transform data from one representation to another. The representation decides the cost.
  2. Access patterns drive layout. Organise data by how and when code reads and writes it together, not by real-world taxonomy.
  3. Structure of Arrays (SoA) over Array of Structures (AoS). Prefer parallel contiguous arrays per attribute over arrays of fat heap-allocated objects.
  4. Separate hot and cold data. Isolate state read or written on high-frequency paths from metadata read rarely or once.
  5. Handles over pointers. Use keys, indices, and offsets instead of per-object heap pointers and object trees.
  6. Batch over per-object. Write functions that process contiguous collections in bulk, not one call per object.

2.2. There is no entity

The rule is section 5.4 of Richard Fabian's Data-Oriented Design (2013), There is no Entity (dodmain/node5.html). The section names the step of turning a class into "containers of smaller objects, an approach often called composition" as a transitional state, and puts the destination beyond it: "A component based object is nothing more than the sum of its parts", and its definition is "an inventory with some construction arguments".

The reading this project takes is the one the trading model wrote down: no object graph is modelled; rows are atomised and keyed; relationships are foreign keys and joins; there is no parent that owns children, and no aggregate versions as a whole. See dfe15809, the trading data-oriented model.

Three consequences bind this project, and all are reviewable.

  • The components are separate types. No aggregate type represents an entity, whether its parts are nested members or a read-time aggregate. Each component has its own type, its own key, and its own codec.
  • There is no entity object, so no entity object travels. A message carries one typed array per component, joined by key. One component's fields never appear as nested objects inside another.
  • The unified view is assembled for display. A screen that needs one row from several components reads the arrays for one page and zips them. The server keeps filtering, ordering, paging and the total, so the arrays stay aligned. A screen that spans several entities is a read model, never a client join; see NATS entity protocol specification.

This is the target for every component, not for one. Codegen does not reach it yet: it emits one aggregate type per entity and projects that type directly, so a grouped entity's members appear nested on the wire, as the trading instrument fixtures do with identity. The atomised shape is a codegen change, and until it lands an aggregate is a known defect against this rule rather than a licence for it. The atomised shape also relieves the translation-unit pressure the C1202 section records, because no unit reflects one wide entity type.

The target shape, the split rule, the part naming and the read are in The atomised entity: a central row and its parts.

2.3. Where each axiom applies

ORE Studio is a database-backed application with Qt clients and a quantitative engine on QuantLib. DOD is not one rule for the whole codebase. A review that demands SoA for a CRUD entity, or accepts a pointer graph inside a simulation kernel, has misread the regime.

2.3.1. Regime 1 — numerical kernels

The hot paths: valuation, simulation, curve bootstrapping, analytics, compute workunits. Axioms 2-6 apply in full. Typical evidence: a loop over trades, paths, or steps that reads a few fields of each object.

2.3.2. Regime 2 — persisted domain data

Entities and their C++ value types: reference data, trading instruments, market data, synthetic entities. The database owns the storage layout; the C++ types are the in-memory shape, and the wire is their flat projection. DOD here means value semantics, identity by key, and a clean split of core state from cold annexes — not cache-line gymnastics.

2.3.3. Regime 3 — wire and messaging

NATS messages, JSON, CSV, and table I/O. The axioms apply as: flat value-shaped messages, versioned schemas, no polymorphic payload fields, and batched messages for high-volume feeds.

A message carries no std::variant field and no std::string field holding an encoded value. A heterogeneous result set travels as one typed array per concrete type, joined by key, so each element's type is static from the array it sits in. The tag belongs to the container, not to every element. See Polymorphic payloads over NATS for the deprecated pattern this replaces.

The wire shape is a codegen projection of the Regime 2 entity, not a second artefact: one entity yields one JSON codec and one table codec. The projection is flat. A grouped C++ type is projected as flat value fields, per There is no entity above; its members are a layout decision for memory and storage, not a wire contract. Review each entity once — its org model, its generated C++ type, and its projections together — and do not walk the message shape again separately. Hand-written message types are the exception; they get their own Regime 3 walk.

2.3.4. The boundary of the regimes

The three regimes cover table-shaped data and its projections. They are a labelled cover of this codebase, not a partition of every possible data shape. One category sits outside them: document-shaped payloads — an FpML trade document, an ORE XML input, an org-mode file parsed into a heading tree. Such documents enter at the boundary of the stack, where a mapping layer converts them to and from the core data-model representation (for example, an FpML trade maps to trade and its instrument entities). The document tree is transient; the entity representation, designed for performance, is the review target.

For a document-shaped type, mark the table-shaped-children and flat-value-message checks as not applicable and say why. In-tree today the category exists only as hand-written core types in ores.orgmode (the heading/document structure); no entity model instantiates it yet. Revisit this paragraph if an entity model ever persists document trees.

A document is not a classification of data on its own. It is an artefact at the boundary, and the data it becomes is classified separately, per the next section.

2.4. Classification precedes the pattern

The three regimes say where an axiom applies. They do not say what a set of data is, and they are not a substitute for asking. A modeller classifies the data first, then chooses the pattern for that classification: Data classifications: entity, list, family, bundle, package and artefact. It is a closed set of six: an entity, a list of one entity kind, a family written in one transaction, a bundle of related families that one owner applies in one unit of work, a package that carries bundles between systems as bytes, and a boundary artefact. The same page names the anti-type — a hand-written union of unrelated entities presented as one struct — and the test that refuses it.

The rule that decides a set of different types is one question: do the families share an invariant that must hold at one instant, or one operation whose intent is all-or-nothing? With no relatedness, split the set into per-family or per-entity messages; if the bytes must still move, they are a package. With relatedness and one owner, use a bundle. With relatedness and several owners, no bundle spans them: use a package carrying one bundle per owner. A file's contents are not an invariant; a file is an artefact.

The form a modeller fills in, and the review artefact, is The data classification checklist.

Classification precedes the pattern, and it also outranks a compile error. A type that is correctly classified and still does not compile has a technical problem; denormalise for the compiler if the shape needs it, and keep the classification. A compile error never reclassifies the data.

2.5. Decomposition and the MSVC C1202 ceiling

The generated codecs have replaced compile-time reflection, so this limit no longer applies to them. It is recorded because the decomposition it forced is still in the models. A reflected struct had a hard shape limit that nothing in the type system expressed, and crossing it failed the build on MSVC with C1202.

MSVC capped the recursive type-dependency graph that reflection built. The count that mattered was the number of field names in one reflected literal, and that literal depended on how the type was written:

  • A plain struct gets a literal of its own members. A nested struct member adds one name, not its members' names. Generated entities with as many as 59 plain members compile on MSVC; on 2026-10-03, 220 of the 287 reflected structs the models generate had more than nine members.
  • Flattening a member adds the flattened struct's names to the parent's literal, recursively. Flatten is a JSON-shape decision, never a decomposition fix.
  • Writing a std::variant with tags builds one literal over every field name reachable from the alternative being written. This is the shape that has failed: the 502-name trade detail wrapper in sprint 19, and the bond export carrier bond_instrument_data on main in October 2026.

The limit reaches C1202 by two paths. One is the width of a single literal, which is why moving a type between headers does not help and no compiler flag raises it on MSVC. The other is accumulation across a unit: a unit that reflects many individually safe literals can still fail. Measured 2026-10-13, the 24-member tenor_basis_swap_convention literal built in a unit of twelve repository headers and failed inside the configuration document handler, which reaches 58 document element types through 103 repository headers.

The rule: keep the names reachable from one tagged write small, by writing the parts of a large carrier separately rather than as one tagged aggregate. Splitting a wide entity into field_group members lowers the parent literal, so it is a remedy for the per-literal path. It is not a wire decision: the JSON stays flat per There is no entity above, and a change that nests it has changed a contract to fix a compile error. No direct-member count is the C1202 limit; the nine-member figure this page once gave was a margin, not a measured ceiling.

The accumulation path gives a second rule for document models. A model that aggregates several entity families hands its unit a literal width equal to the sum of them, so name one bundle per document model and split a document that covers two. A component's per-entity protocols do not help here, because a document protocol includes the domain headers directly and re-aggregates what the per-entity split separated.

That rule keeps a document message that a real invariant sanctions. It does not sanction a message that carries families the document merely lists. Where there is no cross-type invariant, the file is an artefact and each family has its own message, per Data classifications.

2.6. The data model of a component

The artefacts a review reads, per component under projects/ores.<name>:

  • modeling/*.org — the codegen entity, junction, and field-group models (where the component is codegen-driven). The lookup_entity metatype is reserved in codegen but has no authored instances; do not hunt for lookup models.
  • The domain C++ types they generate, or hand-written domain headers in the core facet. This generated representation is the core data model, designed for performance; the review target.
  • The SQL schema the entity maps to: the CREATE TABLE statements generated under projects/ores.sql/create/<component>/.
  • Repository and service signatures: how code fetches collections and looks up records.
  • Kernel entry points and their inner loops, for Regime 1 components.

2.7. Review criteria

2.7.1. Regime 1 checklist

Check Description
SoA on the hot path Fields a kernel iterates sit in parallel contiguous arrays, not in an array of fat objects.
No hot-loop allocation Buffers are reused; no allocate/free or lock inside the loop.
No per-object dispatch The hot path calls batch free functions over spans; no virtual or per-object indirection.
Hot/cold kernel inputs Metadata read once per run is not interleaved with per-step state.
Predictable access Inner loops are stride-1 over contiguous memory so the compiler can vectorise.
Plain inputs Kernels take plain structs and free functions (e.g. a log-normal short-rate SDE), not stateful class hierarchies.

2.7.2. Regime 2 checklist

Regime 2 review walks two surfaces, and each check below states which one it applies to:

  • The decision surface: the modeling/*.org model choices — key style, field-group split, junction versus blob, batched access. These choices can fail; codegen does not rescue them. Review them per entity model.
  • The invariant surface: on codegen entities, value semantics, pointer-freedom, and generated codecs are generator invariants. Codegen cannot emit a shared_ptr or a hand-rolled codec. Walking a guarantee per entity rubber-stamps it. Spot-check each invariant once per component on the generated output, or rely on the codegen drift check (the compass-codegen-fix-drift skill owns drift). Hand-written domain types in the core facet have no generator; walk both tables in full on them.

Decision checks — walk per entity model:

Check Description
Identity by key Relationships are keys and foreign keys, not pointers or owning-parent links.
Table-shaped children Child records are rows in their own table (junction/FK), not nested object trees.
Core/cold split field_group and table design separate core fields from cold annexes (change-reason cache, history, versioning, audit). The split is for memory and storage; the wire stays flat.
Batched collection access Repositories and services return collections, not row-at-a-time callbacks.
Explicit variation A group of related entity kinds (for example, instruments) states its variation explicitly — per-type tables or a closed variant — not open inheritance webs.
One component per row Every column of a row holds for every row. Three shapes break this, and each is a part of its own, keyed by the owner, with at most one row per owner: a has_* flag followed by the columns it governs (the part's presence replaces the flag); a discriminator followed by columns that only one of its values uses (one part per value); two legs written as column sets, each leg with its own terms (one part per leg). Two values of one term, such as a threshold each way of one agreement, are two fields of one component and not two legs. A width is never the argument for a split, and a type with none of the three shapes stays whole, however wide.
Exact money at rest A money quantity — rate, spread, notional, amount, price, strike, fee, premium, margin, face value — is exact decimal in both the domain type and the column, and the two agree. A continuous quantity — volatility, correlation, weight, probability, recovery rate, tranche fraction, day-count fraction, unit quantity — stays binary float.
One-way numeric conversion A double never enters the persisted domain. Conversion runs from decimal to double, at the numerical-kernel seam, and only there. A value that arrives as text is read with a decimal parse, not through a binary float.

Invariant checks — spot-check once per component:

Check Description
Value semantics Generated domain types are plain values with a key; no owning-pointer members.
No shared-ownership webs No shared_ptr inside domain types; ownership sits at component boundaries.
Codec drift JSON/CSV/table codecs come from codegen unmodified; regenerate rather than hand-patch.

2.7.3. Regime 3 checklist

Check Description
Flat value messages Messages are value-shaped, not object graphs over the wire. A composite C++ type is projected flat; its members never appear as nested JSON objects.
No polymorphic payload fields No std::variant field and no encoded std::string field; one typed array per concrete type, joined by key.
Feed batching High-volume feeds batch items per message instead of one message per item.

2.8. Report conventions

A component review produces, per component:

  • A regime classification with the reasons (what is hot, what is cold, where the boundaries are).
  • A findings table: severity (Critical / Important / Minor), effort (Small / Medium / Large), and the criterion each finding maps to.
  • A per-regime score of 1-5 where the regime applies.
  • An overall data-model verdict and a debt level of Low / Medium / High.

Scores are anchored, not arithmetic. Pick the row the review most resembles:

Score What that looks like Debt
5 All applicable checks pass; only evidence-free opportunities remain. Low
4 Decisions pass with Minor findings; invariants hold. Low
3 One Important finding, localised to one model or one kernel. Medium
2 Several Important findings, or one Critical finding in a main flow. Medium
1 Critical findings on identity, core/cold split, or a hot loop; systemic. High

The debt bands follow the score: 4-5 is Low, 3 is Medium, 1-2 is High. A regime that does not apply to the component is unscored, not scored 0. Two reviews of the same component should land within one point of each other; if they do not, the classification or the evidence was unclear, not the scale.

Hot-path claims need evidence: the loop structure, a profile, or a benchmark. Without evidence, record a layout risk as an opportunity, not a defect. A finding that demands a transformation outside the component's regime is a review error, not a finding.

3. See also

Emacs 29.3 (Org mode 9.6.15)