Story: Hotfix: the debug builds never finish
Table of Contents
This page documents a story in Sprint 28. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
1. Goal
The debug legs finish. Every clang-debug run of Continuous and Nightly for the last day ended with
##[error]The action 'Run CTest workflow' has timed out after 320 minutes.
not with a test failure. The build was at 3,378 of 7,663 targets three hours in, so it needed about seven hours. The leg builds in four hours when its compiler cache is warm, and no run since the 07:32 one has had a warm cache.
The cache never warms because it cannot turn over. GitHub gives a repository 10 GB, this project needs four sccache variants, sccache's default cache is 10 GB all by itself, and the prune that would free room ran only after a save that was itself waiting for it:
- A debug cache directory holds 3,938 MB; the repository held 7.31 GB of 10 GB, so the upload could not fit.
actions/cache/saveturns an upload it cannot make into a warning and exits zero, so the step reported success and stored nothing.- The prune step then asked the service whether this run's key existed, found it did not, and failed the job without deleting anything.
- Every later run restored the same stale entry, missed, and timed out again.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent sprint | Sprint 28 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-19 |
3. Acceptance
- Each Linux variant caps its cache, so four variants fit the repository's allowance.
- Space is freed before the save, oldest entries of the same variant first.
- A save that stored nothing warns instead of failing a build that passed.
- The step cap leaves room to save: 350 minutes under the 360 a hosted runner allows.
- Both Linux workflows carry the same logic.
4. Decisions
- Cap each variant rather than raise the repository's cache: the allowance is a property of the plan, and four variants at 2 GB fit with room for vcpkg.
- Free space before the save. The older order was safer against a failed save, and it deadlocks the moment the allowance is the binding constraint, which is the state this repository reached.
- Do not fail the job when a save stores nothing. The build it describes has already passed, and the red job hid that the cache, not the code, had failed.
5. Out of scope
- The Windows and macOS debug legs. Their reds are compile errors, not timeouts, and they keep their own caches.
- The nightly gcc legs' compile errors, which are separate.
6. Result
Both Linux workflows now cap sccache at 2 GB per variant, free room in the repository cache before saving (oldest entries of the same variant first), warn instead of failing when a save stores nothing, and give the build 350 minutes. The deletion logic is exercised against three cache layouts: a warm four-variant set that fits, an empty allowance, and a full allowance holding duplicates.
7. Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Scaffold story: Hotfix: the debug builds never finish | DONE | 2026-10-19 | 2026-10-19 | Story scaffolding rides this task: documents, sprint wiring, and the scaffold PR. Close it before merging that PR. |
| Make the compiler cache turn over inside the repository quota | DONE | 2026-10-19 | 2026-10-19 | Initial task for: Hotfix: the debug builds never finish |