Story: External data reorganisation
Table of Contents
This page documents a story in Sprint 10. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
Goal
Get the project's external data into a tidy, provenanced home and route coding schemes through the DQ artefact pipeline like every other dataset.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent sprint | Sprint 10 |
| Now | Completed 2026-01-24. |
| Waiting on | None. |
| Next | None. |
| Last touched | 2026-01-24 |
Acceptance
- External data physically under
external/with methodology docs. - populate/ split into domain-specific subdirectories with manifests.
- Code generators for flags / crypto / solvaris metadata SQL.
- ISO + FpML coding schemes flow through the DQ artefact pipeline (party schemes excepted).
Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Tidy up external data storage | DONE | 2026-05-19 | 2026-01-23 | Restructure populate/ into domain-specific subdirectories (fpml/, crypto/, flags/, solvaris/); manifest.json files; code generators for flags/crypto/solvaris; relocate external data to external/. |
| Move coding schemes through DQ artefact pipeline | DONE | 2026-05-19 | 2026-01-24 | ISO + FpML coding schemes now flow through dq_coding_schemes_artefact_tbl with preview / upsert / insert_only / replace_all modes; party schemes (LEI, BIC, MIC) remain direct inserts. |
Decisions
- Manifest.json per domain
- provenance is grep-able.
- Party schemes stay direct-insert
- small, manually-defined, and they don't benefit from the artefact-pipeline overhead.
Out of scope
- Live vendor integrations — out of scope.
See also
None.