Task: Populate the equity family

• The routing. Eighteen types reach nine tables: equity_option_instrument (EquityOption, EquityCliquetOption, EquityOutperformanceOption), equity_asian_option_instrument (EquityAsianOption), equity_barrier_option_instrument (EquityBarrierOption, EquityDoubleBarrierOption, EquityEuropeanBarrierOption), equity_digital_option_instrument (EquityTouchOption, EquityDigitalOption), equity_forward_instrument (EquityForward), equity_swap_instrument (EquitySwap, EquityWorstOfBasketSwap), equity_variance_swap_instrument (EquityVarianceSwap), equity_accumulator_instrument (EquityAccumulator, EquityTaRF) and equity_position_instrument (EquityPosition, EquityOptionPosition).
• The writer, service side. write_instrument_batch writes every one of those arrays.
• The pair and index precedents. Commodities and currency pairs are drawn from tenant reference data, so the shape of a catalogue and its scope load is settled.

An equity instrument states underlying_name, documented as the ORE equity Name identifier, and nothing in the tree holds one. The instrument code catalogue is not it: ores_refdata_instrument_codes_tbl holds product codes (EQOPT, EQFWD, EQSWP), which is what maps an ORE product to a trade type, not what an equity trade is written on.

So the family needs reference data it does not have, and it belongs in the data quality pipeline in the way the commodity catalogue did: an entity, an artefact table, a publish function, a dataset and a catalogue, seeded from a source that can be checked.

The source is ORE's own examples. The trade corpus names the underlyings it prices, and they are real: .SPX, .NDX, .STOXX50E, .STOXX50, plus the ISINs and RIC forms the same files state. A catalogue seeded from those is not invented, and a reader can check each row against the file it came from.

• D1 — the catalogue is its own task, as the commodity one was. It carries a schema, a DQ pipeline and a publish, and mixing it with the planner would make one task that cannot be verified in two steps.
• D2 — the payload states one idiom or two. The nine tables are less varied than FX's seven were: an option shape, a forward, a swap, a variance swap, an accumulator and a position. equity_terms carries the union as fx_terms does, and the writer dispatches per table.
• D3 — basket columns are named out of scope unless a source exists. equity_swap_instrument carries four *_json columns for baskets, knockout levels and trigger levels. A population that filled them with invented JSON would be worse than one that leaves them null, so the task states which it does and why.

1. The catalogue. Seed the equity underlyings from the ORE examples through the DQ pipeline, as its own task.
2. The payload. equity_terms and its variant entry, plus the underlying list on the scope.
3. The planner. equity_trade_planner registered under equity, cycling the eighteen routed types.
4. The writer. One arm per routed table.
5. The profile. A config for an entity whose books trade equities.
6. The proof. Nine export cases, one per table, then a population run measuring the type coverage and the share.
7. The record. Measurements, Result, and the gaps this task cannot close.

Every one of these was a value the schema refuses, and every one was found by running rather than by reading:

1. The catalogue was seeded and never published: it had no artefact type, no publish operation and no bundle membership.
2. The aggregate registrar never called the equity underlying registration, so the service answered no request for the list.
3. underlying_kind carried the catalogue's index or share where the vocabulary is equity.
4. The accumulator's payoff_type and target_type are a fixed set, and an accumulator states no target where a TaRF must.
5. A digital states an option type and a strike; a touch states a barrier and neither.
6. The forward's quantity is required above zero and was never set.
7. The swap's return_type is Total or Price.
8. The constituent's check read the catalogues as the writing session, and the population writes as a service session that holds none, so every constituent was refused. The vocabularies are market facts and are now read past tenancy.

The catalogue was made tenant-scoped by copying the commodity precedent. The run shows the cost: a check that resolves a code against a catalogue only works when both share a tenant, and a service-origin writer has no catalogue at all. A vocabulary of market facts belongs to the installation, and the resolver now treats it that way even though the rows are published per tenant.

The commodity catalogue is in the same position as the equity one was: seeded, never given an artefact type, never published. Any family that draws on a catalogue will meet this until it is fixed for all of them.

What the corpus actually holds. Restricting to trades whose type is an Equity one gives 121 trades across 40 files, sixteen of the eighteen routed types, and ten distinct underlying strings. Two of those are RIC-prefixed forms of names that also appear bare, so the catalogue the corpus supports is about eight underlyings:

That count is Equity-typed trades only, and it undercounts the shares. The corpus also prices single names in its total return swaps, which are not Equity typed: AMZN.O, CR.N and MSFT.O, three exchange-listed shares. Adding them gives about eleven instruments, of which four are shares (GB0005405286 as an ISIN, and those three tickers) and the rest indices. So the catalogue is not indices only, and the question is the width of the set rather than its kind.

Two further things follow. The set is small: a population of eighteen types over eleven underlyings means each is traded many times, which is realistic for indices and less so for a book that claims breadth. And the set carries aliases: .SPX and RIC:.SPX are one instrument under two names.

What the catalogue must hold. The instrument states underlying_name and the document's Name is what the engine resolves, so the catalogue's key is the string a trade states. The question the schema has to answer is whether an alias is a row of its own or a second name on one row. The two options:

• One row per string. Simple, and every document value resolves. It also lets the same instrument enter a portfolio twice under two names, which a pricing run would treat as two underlyings.
• One row per instrument, with aliases. The key stays the string a trade states, and a second column names the instrument the alias resolves to. More schema and more seed rows, and a population cannot then double-count.

What the pipeline needs, taking the commodity catalogue as the template — 62 files, of which the hand-written ones are the shape to copy:

1. An entity model under projects/ores.refdata/modeling/, which generates the C++, the TypeScript, the create and drop scripts, the protocol, the repository, the handlers and the shell verbs.
2. A data quality artefact table, generated by the same entity flag the commodity catalogue used.
3. A publish function in refdata_publish_from_dq_create.sql, hand written.
4. A seed populate carrying the catalogue, the dataset and the artefact rows.
5. Wiring into the create, drop and populate indexes — hand maintained, and the three times this work has forgotten it the failure was a database recreate.

Open decisions.

• Aliases. One row per string, or one row per instrument with aliases.
• Scope. A commodity catalogue is tenant scoped and published per tenant. An equity underlying is a market fact every tenant needs, so the same choice has to be made deliberately rather than copied.
• Width. Eight underlyings from a checkable source, or a wider set from a source not yet identified. Inventing tickers is not an option, so a wider set needs a corpus or a published list to point at.

Steps 2, 3 and 4 are done: the payload, the planner and the writer.

• equity_terms states the union of the nine routed tables, with the basket constituents and the periods as lists of their own row types.
• equity_underlying_reference on the scope, loaded from the tenant's catalogue in trading_trade_population_writer.
• equity_trade_planner cycles the seventeen routed codes and refuses an empty catalogue.
• The writer fills one arm per table: the option table serves three codes, the barrier table three, the digital table two, the accumulator two, the swap two and the rest one each. The worst-of swap also states its constituents and its periods.

Four cases prove it: seventeen types appear in a population of seventeen, every slot names a catalogue underlying, a worst-of states two constituents and four periods, and an empty catalogue is refused. The synthetic suite passes 1,733 assertions in 78 cases.

What step 5 needs and does not have

The profile config needs a book chartered to trade equities, and no ACME book is. The seeded books are intercompany loans, FX hedges, vanilla swaps, rates options, CDS and forwards; none is an equity book. Adding one means a new row in acme_uk_books_artefact_populate.sql with its catalogue and dataset, through the same pipeline the equity underlying catalogue took.

Until then the family is proven in code and not in the books, and the export cases and the population run wait on it.

The run is the proof that matters, and it found two gaps in what came before.

1. The catalogue's handlers were never registered. equity_underlying_registrar exists and registrar.cpp declares its function, and nothing called it, so the service answered no request for the list:

generate_trades: failed — no service is handling subject 'refdata.v1.equity_underlyings.list'

registrar.cpp is hand maintained, which is why the aggregate drifted from the entity it was meant to serve. With the call in place the error moves on to the next gap, which is how a wiring fix should behave.

2. The catalogue is seeded but never published. The run now fails on the planner's own refusal:

generate_trades: failed — The tenant holds no equity underlying, so no equity trade can name one. Publish the equity underlying catalogue first.

The seed fills the data quality artefact table and the publish function works when called by hand; the publish_bundle step of the acme_demo profile does not reach it. Adding the dataset to the base bundle made no difference, so the publish path is not bundle-driven for this dataset and the mechanism is still unknown. Commodities are in the same position in this database – their catalogue is empty too – so the commodity population likely never ran through this profile either.

Until publishing is wired, a population cannot depend on a catalogue, and that is the next thing to understand rather than to guess at.

The question was answered by following the path rather than the bundle table. A dataset publishes through three things that must all exist:

1. An artefact type row naming the data quality artefact table, the live table and the publish subject.
2. A publish operation: the refdata service derives the SQL function from the subject's last segment, so refdata.v1.ops.publish_equity_underlyings_from_dq calls ores_refdata_publish_equity_underlyings_from_dq_fn.
3. Membership of the bundle the profile publishes, which is base for the reference data a tenant works from.

The catalogue had the SQL function and none of the three. All three are in place and the catalogue publishes during provisioning; the population run now books equity trades.

Booking then found five refusals, each a value the schema does not accept:

• the accumulator's payoff_type and target_type, which are a fixed set, and the split between an accumulator, which states no target, and a TaRF, which must state one;
• the digital and touch split, where a digital states an option type and a strike and a touch states a barrier and neither;
• the forward's quantity, which the schema requires above zero and the writer never set;
• the swap's return_type, which is Total or Price and not a phrase I invented;
• the constituent's kind, which is the vocabulary – equity – and not the catalogue's index or share classification.

EquityOutperformanceOption is excluded: it routes to the option table, and that table's check admits EquityOption and EquityCliquetOption only, so no row of that type can satisfy it. That is a defect in the check, not in the planner, and it is why the family books sixteen of the seventeen routed codes.

The run now fails inside the basket constituent's check:

ores_trading_basket_constituent_resolves_fn(text,text) line 7

The resolver works when called by hand, and the catalogue's eleven rows belong to the ACME tenant. The write that triggers it runs in the trading service's session, whose tenant is not the catalogue's. So the refusal is correct and the question is who the population writes as: either the catalogue must be visible to that session, or the resolve must read the catalogue of the tenant the trade belongs to rather than of the session. That is a design question about service origin writes, and it is the next thing to settle.