Task: Implement Hotfix: nightly CI failures
Table of Contents
This page documents a task in the Hotfix: nightly CI failures story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Fix each confirmed nightly failure in its own commit, and find the origin of the two uninitialised sends that valgrind reports.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Hotfix: nightly CI failures |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-11-09 |
3. Acceptance
- Every item in the story's acceptance has a commit or a recorded finding.
4. Plan
- Change the shell tests and their templates to expect the bare verb.
- Remove the
constthat makes thestd::movein the trade summary repository redundant. - Suppress the NATS async callback thread error slot.
- Use
time_utils::localtime_safein the cron tests. - Find the origin of the uninitialised send with
--track-origins=yes. - Make the change hub epoch unique within a process.
- Give
nats_connect_errora key function in the library. - Name each HTTP client test as it starts, so the macOS hang names itself.
5. Measurements
Every number this task claims about a tree, with the command that produced it
and the commit it was measured at. A number without both is not a measurement,
because a clean-up record describes a tree and trunk moves under a branch: a
consumer count, a gate's component count and a suite's size all change with no
code changing, and the reader of a bare number cannot tell which tree it came
from. Where the raw output matters, put it in an evidence/ file beside the
task and name that file in the Command column.
| What | Value | Command | Commit |
|---|---|---|---|
| Uninitialised send errors, 2 eventing tests | 0 in each test | valgrind --track-origins=yes on write_engine_settings_change_publishes_an_event and write_curve_publish_publishes_an_event |
8d61ecea8b |
| Shell suites passing with cli 2.3.0 | 16 of 16 | each ores.shell.*.tests with ~[long_running] |
0917b0d297 |
6. Notes
Root causes:
- Shell suites: cli 2.3.0 returns the bare verb from completion.
ores.shellusesCliFileSession, so no tab completion is reachable. The tests and their templates now expect the bare verb. - GCC: a
constlocal made astd::moveredundant in the trade summary repository. - Valgrind MPK: the NATS C library holds one per-thread error slot for its async callback thread. The suppression covers this reachable block.
- Valgrind UMR: the
cpp_nats_integration_testtemplate read*party_ctx.party_id()for party-isolated parents. In two testsparty_ctxis the plainh.context(), so the optional is empty and the read is undefined behaviour. The row got uninitialised bytes as itsparty_id. The template now reads the session party only when writes are party-scoped. TLS hid the origin.PGSSLMODE=disableand a field probe located it. - Windows: the cron tests called
localtime_r. They now calllocaltime_safe. - macOS change stream: two epochs could match on a coarse clock.
make_epochnow increases strictly within a process. - macOS next-fire: the exception missed its catch clause.
nats_connect_errornow has a key function in the library, which anchors its type info. This is the likely cause, and only the macOS nightly can confirm it. - macOS HTTP client timeout: cause not found. The tests now print each name at start, so the next hang names the test.
Neither macOS nor Windows ran locally. The nightly run is the final proof for those platforms.
7. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
8. PRs
| PR | Title |
|---|---|
| #4529 | [ci] Fix nightly failures on Linux, macOS and Windows |
9. Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | Local review: with no session party, two eventing tests keep the generator party_id on seeded parents | cpp_nats_integration_test.cpp.mustache | Accept as designed | Both tests pass and valgrind reports 0 errors. |
| 2 | @claude: make_epoch needs a test for strictly increasing values | change_stream_tests.cpp | Accept | Test added. |
| 3 | @claude: the make_epoch counter can differ per shared library | change_hub.hpp | Decline | The macOS failure is in one test binary. |
| 4 | @claude: nats_connect_error may not be exported | nats_connect_error.hpp | No change | The class has ORES_NATS_EXPORT. |
| 5 | @claude: the test start listener is permanent | net_transport_tests.cpp | Accept later | Remove it after the macOS hang is found. |
10. Result
All local acceptance is met on linux-clang-release-ninja: the shell suites pass with cli 2.3.0, and the two eventing tests report zero valgrind errors. The Windows and macOS fixes and the macOS test-name diagnostic ship unproven until the nightly run.