A test file whose cases are compiled away reports green
Table of Contents
This page is a capture in the inbox bucket of the product backlog — a pre-sprint idea, not yet pulled into a sprint as a story.
1. What
Add a check that fails when a TEST_CASE in a test source never reaches the
compiler. For each *_tests.cpp, count the cases the file declares, preprocess
it with its own flags from build/output/<preset>/compile_commands.json, and
fail when a declared case does not appear in the preprocessed output. It writes
nothing and needs no test run, so it can sit beside the other codegen gates and
run in the same CI job.
2. Why
The hazard is silent in the strongest sense. The translation unit compiles, the link succeeds, ctest runs the suite and reports green, and the only evidence is a case count that nothing compares against the sources.
The measured instance is the ores.service readiness suite. Its three
TEST_CASE=s sat inside =#if defined(BOOST_ASIO_HAS_LOCAL_SOCKETS), and the
guard was placed above every boost::asio include, so the macro was never defined
when the guard was read and the file compiled to nothing. ctest reported 39 cases
where the sources held 42, and the three missing names were exactly the readiness
tests. A clean-standard record then cited that suite as the coverage for
systemd_notify.cpp under V08, so the record claimed verification it did not
have. An independent verifier found it by preprocessing the file with its own
flags and counting what reached the compiler.
The same shape covers #if 0, a guard on a macro that no longer exists, a guard
whose defining include was moved inside it, and a platform guard that is wrong
for the target.
The class is small enough to sweep today. There are 684 test sources in the tree
and three carry a preprocessor conditional at all, two in ores.platform and
both correct.
3. References
projects/ores.service/tests/systemd_notify_tests.cpp– the measured case, fixed by moving the include above and outside the guard.build/output/<preset>/compile_commands.json– the per-file flags the check needs, already generated by the build.projects/ores.codegen/scripts/– where the sibling gates live and where a check would sit.doc/agile/versions/v0/sprint_26/clean-service/task_clean_service.org– items V07 and V08, the two rows the defect falsified.
4. See also
- Unit test conventions – the file and case conventions the check would enforce.
- Component Clean Standard – V07, the item that forbids a test which cannot fail, and V08, the per-file survey that credits a suite with coverage.