Story: Split the ER diagram by group
Table of Contents
This page documents a story in Sprint 27. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
1. Goal
The ER diagram is split into one file per package plus an index, drawn as SVG, and the relationships inside each group come from the SQL rather than a stale map.
2. Problem
ores_schema.puml is one diagram over 463 tables in 21 packages, and it
is 622 KB of PlantUML. Two things are wrong with it, and the second is
the reason splitting is worth doing rather than merely convenient.
2.1. The image cannot be painted
plantuml fails on it with java.lang.NegativeArraySizeException in
EmptyImageBuilder, computing a raster of about 2.68e9 pixels, past a
32-bit pixel count. The script already asks for
PLANTUML_LIMIT_SIZE=131072, which is the cap that lets the product
overflow; a smaller cap scales the drawing, and the drawing is far past
the point where scaling leaves anything readable. The committed
ores_schema.png is therefore stale, and the gap widens with every
merge. The er-diagram check compares the .puml alone, so nothing goes
red.
2.2. The relationships are dead
The model renders with relationships: []. detect_relationships looks
each column name up in FK_TARGET_TABLES, whose values are unprefixed
– iam_accounts_tbl, refdata_currencies_tbl – while every table in
the model carries the ores_ prefix. No lookup can ever match, so the
diagram has no arrows at all: 463 boxes and a wall of columns.
The SQL the parser already reads names the arrows for it. The generated
create scripts carry 336 -- Validate <column> (soft FK to <table>)
comments, naming 82 distinct target tables. Splitting the diagram without
recovering them produces more diagrams that still say nothing.
2.3. What is tracked
| Artefact | Size | State |
|---|---|---|
ores_schema.puml |
622 KB | current, gated by the check |
ores_schema.png |
118 KB | stale since it could last render |
ores_schema.cmapx |
13 KB | from 2026-09-30 |
The PNG and the cmapx carry layout coordinates from whatever PlantUML
build is installed, so they differ between machines (see the memory note
plantuml_er_generate.sh also rewrites ores_schema.png and
ores_schema.cmapx). They are committed anyway.
2.4. Measured: a single package renders
Rendering each package on its own against the current model succeeds. The whole-schema figure is the outlier, and the reason is the 21 packages side by side: both dimensions blow up at once.
| Group | Tables | Pixels | Shape |
|---|---|---|---|
dq_staging |
57 | 20481 x 11198 | 229.3M px |
trading |
99 | 1121 x 31744 | 35.6M px |
refdata |
153 | 1450 x 48528 | 70.4M px |
| whole schema | 463 | fails to paint | about 2.68e9 px |
Measured by filtering packages in the intermediate JSON and drawing each
with plantuml -tpng at PLANTUML_LIMIT_SIZE=131072.
The shape matters as much as the size: a package alone is a strip about
320 px per table tall, so refdata at 153 tables is 48,528 px high and
1,450 wide. It opens, and nobody will scroll it. A vector format per
group is the answer to the shape, and the split is the answer to the
overflow.
3. Options
All six are compatible with the existing three-stage pipeline (parse to JSON, render a mustache template, draw). They differ in where the grouping comes from and in what gets drawn.
| # | Option | Grouping comes from | Cost | Risk |
|---|---|---|---|---|
| A | One diagram per package (21 files) plus an index | the model's existing packages |
low: the template already emits package blocks |
none for overflow, measured; the biggest is a 48,528 px strip, so a raster PNG of it is unusable and only a vector format or a finer group fixes that |
| B | One diagram per hand-grouped domain (about 6-8) | a config file listing groups | medium: someone decides and maintains the groups | each diagram is large; only helps if the groups are smaller than the whole, and they are not much smaller |
| C | One diagram per cluster of the FK graph | community detection over the arrows | medium: needs the arrows first | the grouping changes whenever the schema does, so every diagram redraws on unrelated merges |
| D | Group declared per entity in its model (:diagram_group:) |
the models | high: a flag on every entity, defaulted to the component | the most stable and reviewable grouping, and the most work |
| E | Keep one diagram but drop the columns, one inventory per package that shrinks boxes to table names | the model | low | the schema detail moves elsewhere, or nowhere |
| F | Emit SVG per group and stop tracking the PNG and cmapx | any of A-D | low-medium: change the draw step and untrack two files | PlantUML still has to lay the graph out, so it is not by itself a fix for the biggest group |
Two combinations look strongest:
- A + F, with the arrows recovered: 21 small vector diagrams, no raster
budget, nothing environment-specific in the tree. The index names the
groups and the tables each holds. The two large packages lay out
already, so this needs no sub-splitting to work; whether
refdataandtradingdeserve a finer group is a readability question, not a rendering one. - D + F: the grouping becomes model metadata, so a new table has to say where it belongs, and the diagrams stay stable under unrelated merges.
4. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent sprint | Sprint 27 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-13 |
5. Acceptance
- Relationships appear in the diagrams, inferred from the keys the create scripts state, and FK_TARGET_TABLES is deleted.
- One .puml exists per package plus an index that names the groups and the tables each holds.
- Every generated diagram renders on this machine with no raster overflow.
- The draw step emits SVG, and ores_schema.png and ores_schema.cmapx are removed from the tree.
- The grouping and output format decision is recorded as option A plus F.
6. Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Recover the ER diagram's relationships | DONE | 2026-10-09 | 2026-10-09 | The parser renders no relationships because its FK map lost the table prefix. Infer the edges from the SQL it already reads. |
| Split the ER diagram by package | DONE | 2026-10-09 | 2026-10-09 | One diagram over 463 tables cannot be painted. Generate one per package, plus an index. |
| Draw the ER diagrams as SVG | DONE | 2026-10-09 | 2026-10-09 | The raster PNG overflows and the PNG and cmapx carry the installed PlantUML's layout. Emit SVG and untrack both. |
7. Decisions
The remainder is carried in Gate every ER diagram.
Taken: option A + F – one diagram per package, drawn as SVG, with the PNG and cmapx removed from the tree.
A, because the grouping already exists in the model as packages and
needs no new metadata, and because it was measured: every package draws
on its own while the 463-table figure does not. The 21 packages side by
side are what breaks the raster, not the table count, so splitting
dissolves the overflow rather than moving it.
F, because the split alone leaves the shape problem. A package is a
strip about 320 px per table tall, so refdata at 153 tables is 48,528
px high and 1,450 wide: a PNG nobody can read, and a format with a pixel
budget it can do nothing about. A vector file is zoomable and
searchable, and it has no fixed raster to overflow. It also ends a
recurring cost: the PNG and cmapx carry layout coordinates from whatever
PlantUML build is installed, so they differ between machines and every
regeneration is noise.
The drawn SVGs stay committed, though. The documentation site renders
them: a proj: link to an image exports as an image tag pointing at the
file in the repository, so an image outside the tree is a page that
shows no schema. The PNG and the cmapx go because they are stale and
unrenderable; the SVGs take their place.
What was not taken, and why:
- B, hand-grouped domains: the groups would be little smaller than the whole, so it buys less than A for more maintenance.
- C, FK-graph clustering: the grouping would change whenever the schema does, redrawing unrelated diagrams on every merge.
- D,
:diagram_group:on every entity: the stronger long-term shape, because the grouping becomes authored and stable, but it is a flag on every model and A already works. Revisit if the packages prove to be the wrong seams. - E, an inventory-only big diagram: the detail has to live somewhere, and A and F give it a better home.
One thing all of them need: the relationships. The diagram draws none today, so the split would produce 21 pictures of boxes. That is the first task, not a follow-up.
The capture The ER diagram PNG cannot be rendered at the current schema size raised the same candidates and is superseded by this story.
8. Out of scope
- Writing hand-authored diagrams for individual domains; this is about the generated schema diagram.
- Changing the schema so it fits one picture.