Codegen entity meta-model — SQL
Table of Contents
This page is a segment of the Codegen org-entity meta-model hub, covering
the * SQL section and the ores.sql.schema facet it feeds: bi-temporal
table create/drop, insert triggers, and notify triggers.
The columns it operates on are declared once, shared with C++, on the
Keys and columns page — this page is only the SQL-specific fields
layered on top.
This page splits its fields across two diagrams, per Applied MASD § "The entity model file as a unified literate document":
- Structural — always part of the entity's shape, never toggled —
drawn as a UML class diagram under Flags below.
:tablename:is the one field of this section that qualifies: a required identity field with no default, every entity needs exactly one value. - Non-structural — the authored, optional VMM features of the
ores.sql.schemabundle — drawn as a UVL feature diagram under Projection configuration below.
The :has_parent_id: hierarchy block is the exception. Its three
properties are authored at the entity root, not in this section, so
Hierarchy below covers them here while
Keys and columns models them
with the other structural features.
1. Physical model mapping
| Facet | Description |
|---|---|
| ores.sql.schema | Bi-temporal schema: table create/drop, insert triggers, notify triggers. |
2. Flags
The section always opens with a ** Flags drawer:
* SQL ** Flags ,:PROPERTIES: ,:tablename: ores_refdata_books_tbl ,:END:
| Property | Required | Meaning |
|---|---|---|
:tablename: |
yes | The generated table's name. Convention: <product>_<component>_<entity_plural>_tbl. |
Structural view — the SQL segment's fields, per the split above,
plus its one paste block kind (extra drop statements, below), drawn
as its own class: a block kind's ID is the well-known kind UUID,
distinct from any entity's per-instance implementing block:
(Source: entity_meta_model_sql_structural.puml,
rendered with plantuml.)
The :tablename: flag above (ores_refdata_books_tbl) drives the
generated create table statement's identifier directly:
create table if not exists "ores_refdata_books_tbl" ( "id" uuid not null, "tenant_id" uuid not null, "version" integer not null, -- ... columns from Keys and columns ... );
Future SQL sub-sections (custom triggers, indices) live alongside
this Flags sub-heading.
3. Hierarchy: has_parent_id
Some entities form a tree via a self-referencing soft foreign key.
A book belongs to a portfolio, and a counterparty can have a
parent counterparty. Setting :has_parent_id: true makes the
ores.sql.schema facet
generate a recursive-CTE function that walks that tree, so consumers
don't each hand-write their own recursive query. The C++ facets gate
their matching members on the same feature: get_hierarchy on the
repository and the service, and the protocol and NATS-handler entries
that reach them.
All three properties sit in the entity's root * Flags drawer, beside
:has_tenant_id:. :has_parent_id: is a
structural feature, not one of this section's ores.sql.schema knobs;
its bundle is Keys and columns.
| Property | Required | Meaning |
|---|---|---|
:has_parent_id: |
no | Turns the hierarchy block on. |
:parent_id_column: |
if set | The self-referencing FK column name, e.g. parent_party_id. |
:hierarchy_name_field: |
if set | An existing column used as the display label in the tree. |
The first property turns the block on and the other two name the columns the generated function reads. Set the first without the others and the function names empty columns, so author all three together.
counterparty is one such self-referencing entity. Its model shows
where each property lives. The feature and the two column names are in
the root drawer; the table name is in this section:
* Flags ,:PROPERTIES: ,:schema: public ,:product: ores ,:component: refdata ,:profile: self-referencing-hierarchy ,:parent_id_column: parent_counterparty_id ,:hierarchy_name_field: full_name ,:END: * SQL ** Flags ,:PROPERTIES: ,:tablename: ores_refdata_counterparties_tbl ,:END:
The bound Self-referencing hierarchy profile
assigns :has_parent_id: true, so the model does not state it. A model
with no such profile writes the line itself, in that same root drawer.
Generates a
<product>_<component>_<entity_plural>_hierarchy_fn(p_tenant_id, p_root_id, p_from_root)
recursive-CTE function returning flat (id, parent_id, name) rows —
the subtree rooted at p_root_id when p_from_root is false, or
(resolving up to the real root first) the whole tenant tree when
true.
The generated function only produces flat rows; it does not build a
tree. Turning those rows into an actual tree is generic work that
does not vary from one hierarchical entity to the next, so it is not
generated at all — every consumer instead calls the same hand-written
ores::utility::domain::hierarchy.hpp, whose build_tree() turns a
vector of flat hierarchy_flat_row into a tree of hierarchy_node.
:has_parent_id: does not generate everything a hierarchical
entity needs, though. The self-referencing FK column itself
(parent_<entity>_id) and its insert-trigger validation are still
hand-written today. The hierarchy function does not depend on that
column being generated; it composes independently either way.
4. Projection configuration (variability)
Every property below is, in MASD Variability terms, a feature
(ores.sql.schema bundle) — authored on the model's sql section and read
through in codegen/core.py=/=org_loader.py, never computed. None of
them changes the entity's logical shape, only the generated DDL,
triggers, and rules. Full classification and cross-facet index: ORE
Studio Variability Model § "Bundle: ores.sql.schema" — this section is the
narrative home for these knobs; that page indexes them.
The bundle as a feature model:
| Feature | Type | Default | Effect |
|---|---|---|---|
:current_state: |
bool | false | Table holds one row per key and no history at all: no valid_from=/=valid_to, no GIST exclusion, no temporal checks, no version or audit tail, and the model's own primary key. Because it has no temporal axis it has no history surface either: no history endpoint, no history request/response messages and no history subscription. A current-state entity still validates its tenant_id on insert, and keeps any ** Indexes it declares (without a valid_to predicate). Structural consequence: removes the temporal axis. |
:immutable: |
bool | false | Rows are written once and never changed: a trigger refuses UPDATE, DELETE and TRUNCATE with SQLSTATE 55000, and the repository inserts rather than upserts, so a second write of a key fails. Requires :current_state: and the top-level :read_only:. Structural consequence: lets other tables reference the entity with a real foreign key. |
:system_scope: |
bool | false | Table is system-tenant-scoped: tenant_id defaults to the system tenant and the insert trigger forces it. Structural consequence: changes the tenant_id column default. |
:nullable_tenant_id: |
bool | false | tenant_id is nullable (system records carry NULL); selects the nullable-tenant validation and index variants. Structural consequence: changes tenant_id nullability. |
:extra_checks: |
collection<string> | ∅ | Additional CHECK constraints to emit on the table. |
:extra_delete_sets: |
collection<string> | ∅ | Extra SET clauses in the soft-delete rule. |
:fk_copy_validations: |
collection<struct> | ∅ | FK validations that also copy/denormalise fields from the referenced row. |
:text_code_validations: |
collection<struct> | ∅ | FK validations against a text lookup column. |
:party_id_from_book_id: |
struct | absent | Derive and validate party_id=/=portfolio_id from book_id in the trigger. |
:party_id_from_parent: |
struct | absent | Derive party_id from the row named by parent_column in parent_table, in the trigger, and ignore what the caller sent. |
:party_id_from_session: |
bool | false | Set party_id from the session GUC app.current_party_id. |
:rls_tenant_isolation: |
bool | false | Emit enable row level security + a permissive tenant-isolation policy (T1) as a trailer of the create file, and its drop in the drop file. |
:rls_party_isolation: |
bool | false | Also emit a party_id-scoped as restrictive policy (T3) over ores_iam_visible_party_ids_fn membership; requires :rls_tenant_isolation:. |
:rls_system_tenant_visible: |
bool | false | Emit the a system-tenant session sees every tenant widening — tenant_id = current or current = system. Requires :rls_tenant_isolation:, and every use is listed by check_rls_system_tenant_widenings.py; read the ownership classes below before reaching for it. |
:rls_own_or_system_tenant_rows: |
bool | false | Emit the shared-reference and installation-scoped policy shape — tenant_id = current or tenant_id = system, with writes confined to the current tenant. Requires :rls_tenant_isolation:, and is mutually exclusive with :rls_system_tenant_visible:; the guard does not flag it, because it admits only the system tenant's own rows. |
:rls_policy_style: |
enum | default |
Selects the emitted policy shape. default is the shape every tenant-scoped table emits: the ores.iam session functions, called once for each row the policy filters. hot_path emits the same two policies with the session setting read through a scalar subquery, so the setting is evaluated once for each query: it requires :rls_tenant_isolation: and :rls_party_isolation:, and refuses :rls_system_tenant_visible:, :rls_own_or_system_tenant_rows: and :rls_service_read_role:, whose clauses the style would not emit. See below. |
:writer_role: |
psql variable | absent | Emit grant insert on <table> to :<role>; after the policies, for the one role an ingest writes through. The role is named as a psql variable, because the grant is a psql script and the role name is deployment state. The generator checks the name against the same rule :rls_service_read_role: uses. What the role is allowed beyond INSERT – BYPASSRLS, the ownership of the table, the SELECT it does not have – is stated where the roles are created (projects/ores.sql/recreate_database.sql), not here. |
:no_audit_columns: |
bool | false | Drops the version column and the audit tail (modified_by, performed_by, change_reason_code, change_commentary), which a series table cannot carry at its volume. Keeps valid_from=/=valid_to, so the entity keeps its history surface. Mutually exclusive with :immutable: and :current_state:, which drop the temporal axis as well. |
:hypertable: |
bool | false | Makes the table a TimescaleDB hypertable: emits the do $$ block that calls create_hypertable when the extension is installed and falls back to a plain table when it is not. Suppresses GIST exclusion, because a hypertable has no exclusion constraint, and keeps tenant_id out of the primary key, because every unique index of a hypertable must carry the partition column. Requires :hypertable_partition_column:; the generator refuses a hypertable without one rather than emit a create_hypertable call with no column to partition on. |
:hypertable_partition_column: |
column | required by :hypertable: |
The column create_hypertable partitions on. It is normally the series' own time column, and it joins every unique index and the primary key of a hypertable. |
:hypertable_chunk_interval: |
interval | none | The chunk width create_hypertable is given, as the text an interval literal carries without the word interval: 1 day, 30 days. Nothing defaults it, so every hypertable model states one: a model that leaves it out renders interval '', which PostgreSQL refuses. |
:hypertable_retention_interval: |
interval | absent | The age at which a retention policy drops a whole chunk, in the same interval form. Emitted only where the installed licence allows a retention policy; without one the block reports the interval and the service's own job drops the chunks that have aged out. |
Current-state is a distinct axis from :no_audit_columns:. A
:no_audit_columns: entity (a TimescaleDB hypertable) drops only the
version column and the audit tail; it keeps valid_from=/=valid_to as its
transaction-time window, and its reads still filter on valid_to. A
:current_state: entity drops the whole temporal axis: the SQL create has no
valid_from=/=valid_to, no GIST exclusion and no temporal check, and its
primary key is the model's own key (for login_info, account_id), not
(tenant_id, key, valid_from, valid_to). It has no history surface: the
service declares no get_<entity>_history, the C++ and TypeScript protocol
headers state no history request/response pair and no history subject, the
NATS registrar subscribes to no history subject, and the shell-command and
presentation history facets omit the history verb and field mapper. The C++
domain type then has no recorded_at and the repository upserts
(insert_or_replace) and hard-deletes instead of closing a validity window, so
a caller must not expect version rows from read_all or a soft delete from
remove. The insert trigger still
validates tenant_id, and the hand-written table's indexes and constraints
carry over through the ** Indexes and ** Checks sections (plus the
per-column :skip_uuid_check: / :skip_empty_check: and per-FK :skip_check:
suppressions), which is how the iam_login_info table keeps
login_info_locked_idx and avoids gaining a nil-UUID check it never had.
The feature is authored in the * SQL ** Flags drawer, beside the other SQL
projection knobs above.
5. Time-series tables
A series table holds facts that arrive many times each second, so it takes the
four :hypertable*: knobs together and drops the audit tail:
* SQL ** Flags ,:PROPERTIES: ,:tablename: ores_marketdata_ticks_tbl ,:hypertable: true ,:hypertable_partition_column: ts ,:hypertable_chunk_interval: 1 day ,:hypertable_retention_interval: 7 days ,:no_audit_columns: true ,:END:
The partition column is the series' own time column, and it is what
create_hypertable chunks on. A hypertable that names none is refused when the
model is read, because the alternative is a create_hypertable call with an
empty column and a primary key built around it, which fails when the schema
loads rather than when the model is written. ts is the column in the example
above; observation_datetime and fixing_date are the columns in the two
market-data tables that carry a series.
The retention interval is optional but a hypertable without one grows without
bound. Its block is licence-aware: where the installed TimescaleDB reports the
timescale licence the generated do $$ block sets the policy, and a community
edition, which has no retention job, reports the interval instead of ignoring
it. The block that both run is the same, so a table's retention is one statement
whether or not the licence allows it to run.
A hypertable carries no GIST exclusion and holds tenant_id out of its primary
key, because every unique index of a hypertable must carry the partition column.
:no_audit_columns: is what drops the version column and the audit tail from
it; :current_state: would drop the temporal axis as well, which a series keeps.
5.1. Which policy style a table takes
:rls_policy_style: names the shape of the two policies the create file ends
with, and the choice is about what the policy costs:
defaultdiffers from nothing: it is the shape every tenant-scoped table already emits, and it callsores_iam_current_tenant_id_fnandores_iam_visible_party_ids_fnfor each row the policy filters.hot_pathemits the same two policies against the session settings directly:tenant_id = (select nullif(current_setting('app.current_tenant_id', true), '')::uuid)andparty_id in (select unnest(nullif(current_setting( 'app.visible_party_ids', true), '')::uuid[])). The scalar subquery is what makes the difference: the planner evaluates it once for each query rather than once for each row, so a table that filters millions of rows pays for the setting once.
The hot-path shape is a pair, so both isolation flags are required. It also
forces the table's own row-level security and names the party policy select
alone, restrictive, with no null passthrough: a session that states no visible
parties reads no rows, which is the rule an ingest session obeys. It therefore
refuses :rls_system_tenant_visible: and :rls_own_or_system_tenant_rows:,
which name different policies, and :rls_service_read_role:, whose
cross-tenant read lives inside the default body and would otherwise stop
appearing without a word.
A soft foreign key states which tenants it may match. By default the
insert trigger checks the referenced row in the writer's own tenant, so a
row cannot point at another tenant's row. :use_system_tenant: true names
a target that lives only in the system tenant – the platform-managed
catalogues every tenant reads. :system_tenant_visible: true names a
global registry whose system-tenant rows every tenant may read while a
tenant may still own rows of its own, so the check admits the row's own
tenant or the system tenant. Use it for a key into an entity whose
repository declares :system_tenant_visible: and whose row-level policy
exposes system rows, so the trigger and the read path agree; a mismatch
rejects a reference the reads happily return. A junction that carries the
registry's own rows – compute's app version platforms, whose package every
tenant dispatches against – declares the same flag in its * Repository
drawer, which widens its reads the same way. :use_no_tenant: true names
a target with no tenant_id at all. Neither :use_system_tenant: nor
:system_tenant_visible: combines with :enforce:, since a REFERENCES
constraint can only match the row's own tenant.
The check reads the target's current version by default, because a temporal
target holds closed versions beside the live one. A target with no validity
window at all – a hand-written table with no modeling org – states
:target_no_validity: true, and the check then names the key column alone
rather than filtering on a valid_to the target does not have. A target
whose model states :immutable: needs no such flag: the generator reads its
immutability itself.
A foreign key into an immutable entity can be enforced by the
database. A foreign key is normally a trigger check against the target's
current row, because a temporal target holds closed versions that a
REFERENCES constraint would accept as current. An immutable target has
no closed versions, so a foreign key declared with :enforce: true in the
* Foreign keys section renders a named REFERENCES constraint
(<table>_<column>_fk) and no trigger check. A key that matches its own
tenant, the default, names tenant_id on both sides, so a row cannot
reference another tenant's row; a tenant-scoped immutable table therefore
keys on (tenant_id, <primary key>). A key with :use_no_tenant: names one
column and needs a target without tenant_id. The generator refuses
:enforce: when the target table's model is unknown or not immutable, when
the key uses :use_system_tenant:, when the tenant scopes of the two tables
differ, and when the constraint name is longer than PostgreSQL's 63
characters.
A pinned key holds a copied fact to its source. A row may copy a fact
from another row, such as an instrument copying its trade's type, only
when a key holds the copy to its source. A ** Pinned keys table under
* SQL lists, per pin, a name, the copied columns, the source table,
the target_columns they must equal, and an error_message whose % is
the first column. A pin matches the row's own tenant when the row has one,
and the two tables must agree on being tenant-scoped. Against an immutable
source the pin is a constraint, <table>_<name>_pin, and the source must
declare the referenced columns as its primary key or as a unique index with
no predicate. Against a temporal source the insert trigger checks the
source's current row; it holds at insert only, so copy only facts that
never change on the source. The generator cannot check that rule; it is a
modelling rule. In both modes a row whose pinned columns hold a null is
not checked, as PostgreSQL does for a constraint. A pin into a mutable
current-state table is refused, and so are repeated pin names, columns
either model does not declare, and a message without exactly one %.
A routed column takes its codes from a catalogue. A model whose
* SQL ** Flags drawer declares :routed_by: (a catalogue model's title,
such as ores.trading.trade_type_catalogue) and :routed_column: (one of
its own columns) gets a check that the column holds one of the codes the
catalogue routes to this entity, in the catalogue's order. The list is the
catalogue's, so every table's list comes from one source and no code can be
accepted by two tables. The generator refuses one flag without the other, a
column the model does not declare, a title that names no model, and a
catalogue that routes no code to the entity.
A fixed column keeps its first value. A column marked :fixed: true
in * Columns keeps the value of its row's first version: the insert
trigger refuses a new version that changes it, with SQLSTATE 23514, and
it does so for a version replace too. A copy pinned to a temporal source
stays true only when the copied columns are fixed. A fixed column needs a
temporal table; a current-state row has no versions to compare.
Tenant-scoping is a policy, not three booleans. :system_scope: and
:nullable_tenant_id: are mutually exclusive with the default
(standard-tenant) mode; together they form a single enumerated feature —
tenant-scoping policy ∈ {standard, system, nullable}. If they migrate
anywhere, it is into the logical model as a tenant-scoping attribute
(they lean structural despite being authored as knobs).
Name the row's owner before choosing its policy. Every row belongs to a tenant, to the system tenant, or to the installation — the deployment's own facts, which no tenant owns: the services roster, hosts and wrapper nodes, grid and node samples, telemetry logs, the database and build row. The owner decides the policy, and the three are not interchangeable:
| Owner | Policy the table wants | How a model states it |
|---|---|---|
| A tenant | Own tenant only (T1). | :rls_tenant_isolation:, the default. |
| The system tenant: shared reference | Own tenant or system-tenant rows — every tenant reads the system's, and writes stay its own. | :rls_own_or_system_tenant_rows:. |
| The installation | The same shape: the system tenant appears as a constant, not an owner, so the rows carry no tenant's data. | :rls_own_or_system_tenant_rows:. |
| A tenant's data read from outside it | None. Act inside the tenant. | tenant_context::with_tenant, or internal impersonation. |
:rls_system_tenant_visible: is not how a table says it is installation- or
shared-scoped. It emits a different shape — a system-tenant session sees
every tenant — which admits one tenant's data to another and is exactly what
check_rls_system_tenant_widenings.py exists to stop. The two shapes spell
differently and mean differently: tenant_id = system admits only the
installation's own rows; current = system admits everybody's. Read
Tenant ownership and the information flow
before widening or hand-writing either.
Why the flag is named :rls_own_or_system_tenant_rows:, and not after
the classes it serves. The old hand-written idiom and the new flag must
be told apart by name alone, because a reader who confuses them reaches
for the flagged widening. :rls_system_tenant_visible: names the
session — it is the system tenant that sees more — and emits current =
system. The new flag names the rows it admits and keeps tenant_id
on the left of both comparisons: tenant_id = current or tenant_id =
system. So the name says what is admitted (the row's own tenant, or the
system tenant's own rows) rather than who is asking, and it can never be
read as a widening. It serves both ownership classes that want this
shape, shared reference and installation, because they want the same
rows and the same write rule; naming it after one class would invite a
second identical flag for the other.
Row-level security emission is an explicit intent flag, not a
consequence of having a tenant column. The :rls_*: features above
emit enable row level security and the matching policy blocks inline
at the end of the generated per-entity create file, so a regenerated
table owns its whole SQL lifecycle. They default off and no profile
fixes them: many tenant-column tables deliberately skip RLS
(system-owned registries, currently silenced by RLS_001 ignore
entries in validation_ignore.txt), and a party_id column is not
party isolation scope on its own (e.g. market_series). The emitted
blocks are verbatim copies of the dominant hand-written shapes from
the components' *_rls_policies_create.sql files (T1 tenant-only, the
own-or-system-tenant shape, the :rls_system_tenant_visible: widening,
T3 restrictive party membership). Because
create policy resolves ores_iam_current_tenant_id_fn at creation
time, inline emission is only legal for components created after the
iam section of create/create.sql; the pre-iam components keep their
RLS-layer files until that ordering question is resolved.
SECURITY DEFINER on insert/validate triggers is not a feature —
checked against library/templates/sql_schema_domain_entity_create.mustache, it
is emitted unconditionally today, matching the
entity commissioning process reference's mandatory rule. It is
mentioned here only so nobody goes looking for a
:security_definer: knob that no longer exists.
6. Paste blocks
The SQL segment defines the following paste block kinds, one per
sub-heading below — see Paste blocks for the general mechanism they
assume.
6.1. Extra drop statements
This paste block defines additional DROP statements inserted into
the generated <component>_<entity_plural>_drop.sql, just before
drop table if exists. Use this when the entity's companion
_table.org model generates database objects (e.g. a validate
function) that the entity template itself has no way to know about.
If no entity implements this kind, the marker collapses to nothing.
Example of an inclusion of this paste block in a template (sql_schema_domain_entity_drop):
drop function if exists {{product}}_{{component}}_{{entity_plural}}_insert_fn;
<<paste:5E47F108-1350-4540-B3C2-E83DD5379B2D>>
drop table if exists "{{product}}_{{component}}_{{entity_plural}}_tbl";
To use this paste block on an entity, declare it under ** Paste
blocks > *** Extra drop statements, with the body in a block named
body. See the following example:
** Paste blocks *** Extra drop statements #+begin_src sql :name body :implements 5E47F108-1350-4540-B3C2-E83DD5379B2D drop function if exists ores_refdata_validate_rounding_type_fn; #+end_src
An entity is not limited to a single drop statement, nor to a
single block: it may write several statements inside one body
block, or spread them across several *** Extra drop statements
sub-headings. Either way, codegen collects them all, in the order
they appear in the model, and concatenates them with a blank line
between each.
7. See also
- Codegen org-entity meta-model — the hub.
- Keys and columns — the column declarations this section's DDL is built from.
- Paste blocks — the shared paste-block mechanism.
- Variability — MASD's formal vocabulary for the
ores.sql.schemafeatures above.:tablename:(§ "Flags") is structural, not variability.:has_parent_id:(§ "Hierarchy") is variability, but its bundle is structural. - ORE Studio Variability Model — the
ores.sql.schemabundle's classification and cross-facet index; this page is the narrative home for those knobs.