ctest can report green when a Catch2 binary fails to register its cases

Table of Contents

This page is a capture in the inbox bucket of the product backlog — a pre-sprint idea, not yet pulled into a sprint as a story.

1. What

Make the component test suites fail on a Catch2 registration fault. Either drop --filenames-as-tags from the common argument list and give the tag filter some other source, or add a cheap second invocation per suite that lists the cases and fails on a registration error. A gate in the build, not a habit, is what makes this safe.

2. Why

A generated file per entity makes duplicate case names a live hazard. Every unit of one component lands in one test binary, and two units can pick the same name without either author seeing the other. Catch2 rejects the duplicate when it enumerates the cases, which is the work --filenames-as-tags lets it skip. The result is a suite that reports 100 per cent passed while it is running a different set of cases than its source declares, or none at all. The failure is silent in the direction that matters: ctest is the gate, and the gate says green.

The measured case: the first draft of the shell-command tests archetype reused a hand-written file's case names verbatim. ctest passed and reported 16 cases, while --list-tests on the same binary exited non-zero. Once the names carried a per-entity prefix, all three agreed on 16 cases and 30 assertions.

3. References

  • projects/CMakeLists.txt:76 — the shared argument list, with --filenames-as-tags and --allow-running-no-tests.
  • Every projects/ores.*/**/tests/CMakeLists.txt — two add_test registrations per suite, both passing ${ORES_CATCH2_ARGS}.

4. See also

Emacs 29.3 (Org mode 9.6.15)