Task: Let the convention publish parse
Table of Contents
This page documents a task in the Atomise every convention onto the central entity — the remaining work story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Make db recreate succeed again: the convention publish function must be valid SQL.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Atomise every convention onto the central entity — the remaining work |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-30 |
3. Acceptance
- A comment no longer breaks the string concatenation of the insert statement.
- compass db recreate -y -k completes.
4. Plan
1. Move the block comment out of the literal run in ores_refdata_publish_conventions_from_dq_fn. 2. Teach check_sql_hygiene.py to see a comment between two literals, so the next one fails the sql-hygiene check instead of a database recreation. 3. Recreate the database and publish the corpus twice: the first publish inserts, the second skips.
5. Measurements
Every number this task claims about a tree, with the command that produced it
and the commit it was measured at. A number without both is not a measurement,
because a clean-up record describes a tree and trunk moves under a branch: a
consumer count, a gate's component count and a suite's size all change with no
code changing, and the reader of a bare number cannot tell which tree it came
from. Where the raw output matters, put it in an evidence/ file beside the
task and name that file in the Command column.
| What | Value | Command | Commit |
|---|---|---|---|
| Split literals in the tree | 0 | python3 projects/ores.sql/utility/check_sql_hygiene.py projects/ores.sql |
this branch |
| Split literals in main's file | 1, at line 5268 | python3 projects/ores.sql/utility/check_sql_hygiene.py on the origin/main copy |
769522285a |
| Files the check scans | 2955 | as above | this branch |
| Artefact parameter rows, stale database | 2283 | compass sql -- -c "select count(*) from ores_dq_ore_parameters_artefact_tbl" |
dd67116de2 |
| Artefact parameter rows, after a recreate | 2295 | as above | 769522285a with this branch |
| Database recreation | complete, 1m 11s | compass db recreate -y -k |
this branch |
| Published parameters, first call | 2295 inserted | ores_reporting_publish_ore_parameters_from_dq_fn, rolled back |
this branch |
| Published parameters, second call | 2295 skipped | as above | this branch |
| Value domains the publish needs first | 5 inserted | ores_reporting_publish_ore_parameter_value_domains_from_dq_fn |
this branch |
Parameters whose XPath is under /Conventions/ |
248 | select count(*) ... where xpath like '/Conventions/%' |
this branch |
6. Notes
The defect came in with d1f8da3523, which wanted a republish of a convention
to be a skip rather than a duplicate key. It appended on conflict do nothing
to the insert the function executes, and wrote a block comment between the two
adjacent literals. PostgreSQL joins two constants separated only by whitespace;
a comment is not whitespace, so the second literal became an argument. Nothing
ran the function between the merge and this discovery: setup_schema.sql failed
at the file, so compass db recreate -y -k aborted with exit 3.
The guard lives in check_sql_hygiene.py, which already scans the SQL tree for
a comment that breaks a recreation. It now also scans inside dollar-quoted
function bodies, because a body is SQL the server runs.
7. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
8. PRs
9. Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | A quoted identifier ended on its first quote, so a doubled quote resumed the scan inside the name | projects/ores.sql/utility/check_sql_hygiene.py |
Fixed in this PR | One helper now ends both a string and an identifier on a lone quote |
| 2 | The check costs about 63s where it cost about 49s, because it reads the tree twice | build/scripts/local_checks.py |
Accepted | The second read is the rule that stops the next broken recreation |
10. Result
compass db recreate -y -k completes again, and the function publishes as its
commit intended: 2295 parameters inserted, and the same 2295 skipped on the
second call. The comment sits above the statement, so the literals run together
and the insert keeps its on conflict do nothing.
The sql-hygiene check now reports the defect that main shipped. On main's own file it names line 5268; on this branch it reports nothing across 2955 files.