Every catalogue in the platform is seeded with the same boil…
Table of Contents
This page is a capture in the inbox bucket of the product backlog — a pre-sprint idea, not yet pulled into a sprint as a story.
1. What
Every catalogue in the platform is seeded with the same boilerplate data quality metadata, so none of it says where the data came from or what it is worth.
2. Why
Both the commodity and the equity catalogue record: methodology 'OreStudio Code Generation Methodology', source system ORESTUDIO, licence 'Internal Use Only', business context 'Seed data for the <x> Librarian bundle', as-of date the day the seed ran, coding scheme NONE, origin Primary, nature Actual, treatment Raw.
Almost every one of those is wrong for these two datasets:
- The equity catalogue is not generated by codegen. It was extracted from the ORE example corpus by reading the Underlying elements whose Type is Equity, and its rows carry three different identifier schemes at once: RIC (RIC:.SPX), ISIN (GB0005405286) and exchange-suffixed tickers (MSFT.O). A dataset-level coding scheme of NONE is therefore false at the row level.
- The source system is the ORE examples, not OreStudio. Nothing records which file a row came from, so a reader cannot check a single row against its origin.
- The currency is not stated by the documents. It was assigned from each instrument's home market, which is an editorial decision and is recorded nowhere.
- The licence is filler. ORE ships under its own licence and 'Internal Use Only' is a claim nobody made.
- The as-of date is the seeding date, not the corpus vintage.
- Commodities have the same problem: the exchange-prefixed codes came from ORE's own index corpus, and the dataset says it came from codegen.
The DQ model already has the fields for this: coding_scheme_code, origin_code, nature_code, treatment_code, methodology_id, source_system_id, business_context, upstream_derivation_id, lineage_depth, as_of_date, license_info. The requirement is that a catalogue states them truthfully, that the identifier scheme and the source document are recorded per row where they differ, and that an editorial decision such as assigning a currency is recorded as a decision rather than silently applied.
Two things are needed beyond the fields: a source system for the ORE corpus, and a place to record a per-row origin, which the artefact tables do not have today.
3. References
- Recovered from commit 2e346d7431 on branch hotfix/publish-the-commodity-catalogue.