The data tiers
Table of Contents
1. Summary
The data an installation holds divides into five tiers, and the tier decides nearly everything else about it. The axis is substitutability: how many valid instances of the data exist. A vocabulary the platform defines has exactly one. An external standard has one because the platform adopts a source. Installation data has as many instances as there are installations. Market data has one for every observation time. Transactional data has one for every act of use.
The tier answers the two questions a developer actually has: must the system ship these rows, and who decides which values are right. It does not replace the classification the DQ model already carries. A dataset still has a domain, a subject area and a catalog, and a value still has a nature and an origin. Those say what the data is. This says how firmly the platform holds it.
2. Detail
2.1. The axis
Substitutability is how many valid instances exist. It decides delivery because the question "may this vary by installation" has a different answer at each level, and a table that looks like its neighbour can sit a tier away from it.
| Tier | Name | Valid instances | The system ships it |
|---|---|---|---|
| 1 | Platform vocabulary | One | Always, with the system |
| 2 | Canonical reference data | One, adopted from a standard | Always, as the canonical dataset |
| 3 | Installation data | One per installation | Only as an exemplar |
| 4 | Market data | One per observation time | As samples, or not at all |
| 5 | Transactional data | One per act of use | Never |
2.2. Tier 1: the platform's own vocabulary
These are the codes the platform defines so that it can function at all. A trade link type, a trade activity type, a trade state, a code domain or a business process has no meaning outside this system, and an installation that chose its own values would break it. There is exactly one correct set, and the platform is its author.
A new table at this tier is usually the last step of a domain decision rather than a data decision: the vocabulary was settled in a knowledge page or a story, and the table is where it lands.
2.3. Tier 2: canonical reference data
The values come from outside, and more than one source could supply them. The platform adopts one source as canonical, records which, and expects every installation to hold the same values. ISO 4217 currencies, ISO 3166 countries, the FpML coding schemes and business centres, and the GLEIF LEI records are all tier 2.
The choice of source is the decision here, not the values. Once the source is named, an installation has nothing to choose.
2.4. Tier 3: installation data
The business entities of one installation: its accounts, books, portfolios, netting sets and CSAs. Every installation has its own, and any valid instance exercises the system as well as any other. ACME is an example installation, not the installation.
The system may ship an exemplar so that a fresh installation has something to show, and that exemplar is disposable. Nothing depends on its values.
2.5. Tier 4: market data
Observations of the market, one per observation time. A live feed, a curated vendor set such as the Federal Reserve H.10 rates, a synthetic feeds dataset and the ORE samples are all tier 4. What separates them is the value's nature and origin, not the tier, and the platform already states those two dimensions.
2.6. Tier 5: transactional data
What use produces: trades, amendments, activities and links. The system never ships it, no dataset supplies it, and it exists only because somebody did something.
2.7. What follows: the delivery mechanism
The tier, not the component and not the table's shape, chooses the machinery.
| Tier | How the rows arrive |
|---|---|
| 1 | A populate script under projects/ores.sql/populate/, idempotent, written for the system tenant |
| 2 | A DQ dataset, published into the owning service and installed as a bundle |
| 3 | An ores.seeder dataset |
| 4 | A DQ dataset for a curated set, a synthetic feeds dataset for generated data, or a feed |
| 5 | Runtime alone |
2.8. The procedure
Ask these in order and stop at the first yes.
Does the platform define these values so that it can function? Tier
- Write a populate script. The page that settled the vocabulary owns
the values.
- Do the values come from an external standard? Tier 2. Name the canonical source on the dataset, then treat it as tier 1 for delivery.
- Is this an entity belonging to one installation? Tier 3. Add it to a seeder dataset, and expect it to differ everywhere.
- Is it an observation of the market at a time? Tier 4. Choose a curated dataset, a synthetic feeds dataset or a feed.
- Otherwise it is produced by use. Tier 5. Add nothing; the write path is the delivery.
The last question catches the tables that need no seeding at all, which is the most common mistake in the other direction: seeding transactional data to make a screen look populated.
2.9. What this page does not cover
This page says which data exists, who owns its values and how the rows arrive. It says nothing about where the bytes live, which is the storage cluster's subject.
- Data storage — the storage cluster's structure note: the relational database, object storage, the durable stream and the caches, and which of them owns what.
- Object storage — the bucket-and-key contract, the HTTP interface for bytes and the NATS interface for commands, over a replaceable backend.
- Storage policy — the policy for the installation's own data, and why it is the installation's.
- ores.storage and ores.storage.objects — the service and the entity that records a stored artefact.
The two subjects meet at one point. A tier-2 dataset and a tier-4 sample both end up as bytes somewhere, and which bucket holds them is the storage pages' answer, not this page's.
3. Sources
The tiers are this page's synthesis, drawn from the classification the DQ model already carries and from the delivery mechanisms the tree actually uses. The words are this page's choice; no external standard fixes them.
4. See also
- Data quality — the model this page builds on: domains, subject areas, catalogs, natures and origins.
- ores.dq.data_domain — the high-level classification, and where a tier-4 dataset is told apart from a tier-3 one.
- ores.dq.nature_dimension and ores.dq.origin_dimension — how a tier-4 observation came to be and where it entered.
- Publication pipeline — how a tier-2 dataset reaches the service that owns the table.
- ores.seeder — the machinery for a tier-3 exemplar.
- ores.dq.artefact_type — the endpoint a tier-2 or tier-4 dataset publishes through.
- Trade link types — the worked case that put this page on paper: a vocabulary the platform defines, and therefore tier 1.
- Netting sets — tier 3, and an example of a domain object whose rows belong to one installation.
- Knowledge — the hub of all knowledge notes.