Story: Canary CI: near-zero sccache hit rate makes every build a cold rebuild

Table of Contents

This page documents a story in Sprint 22. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

Goal

Diagnose and fix why canary-linux.yml (linux-gcc-debug-ninja) gets almost no benefit from its sccache cache, so canary stops taking ~3h/run for small PRs.

How this was first noticed (corrected)

Originally filed as "the CTest step doesn't respond to GitHub Actions cancellation" — PR #1410's canary job appeared stuck in_progress for 45+ minutes after two gh run cancel calls. That framing was wrong: the job wasn't stuck. It eventually finished on its own with conclusion: success after a genuine ~3h1m59s runtime — GitHub Actions doesn't preempt a running job mid-step; a long single-step job (Run CTest workflow) simply keeps running to completion regardless of a cancellation request against a superseded run. There is no bug in cancellation handling to fix.

The real question the incident surfaced: why does a PR touching ~20 files take 3 hours to build at all when sccache is configured and a full cache was restored? That's this story.

Evidence: sccache stats from PR #1410's canary run

From the completed run's sccache --show-stats step (gh run view --log --job=84963313660): 

Compile requests                   2433
Compile requests executed          2433
Cache hits                           33
Cache misses                       2400
Cache hits rate                    1.36 %
Non-cacheable compilations            0
Cache size                            2 GiB
Max cache size                        2 GiB

A full 2018 MB sccache archive was successfully restored at the start of the job (~15s download over the Actions cache), and every single compile was cache-eligible (Non-cacheable compilations: 0). Yet 2400/2433 files were recompiled from scratch, despite the PR diff touching only ~20 files. Ruled out:

  • Precompiled headers: this GCC canary preset doesn't use PCH at all (0 matches for include-pch=/=cmake_pch in the full job log — that pattern only exists in a different, local clang preset).
  • Cache eviction from being at the size cap: plausible contributor (2 GiB for a ~5,300-translation-unit monorepo is tight), but doesn't by itself explain why unchanged files miss — sccache keys are content hashes, not identity, so an unrelated PR's build shouldn't evict entries this PR needs unless the working set genuinely exceeds 2 GiB.
  • "PRs overwrite each other's cache" as the direct cause: initially suspected (three PRs' canary jobs were running concurrently, #1408/#1409/#1410, all reading/writing the same static key sccache-linux-gcc-debug-ninja), but overwriting alone doesn't explain a low hit rate — sccache hits are keyed on preprocessed content, not on which run wrote the cache. Concurrent writes could matter only via GitHub Actions cache's own eviction/immutability semantics (see workflow notes below), not directly.

Working hypothesis: dependency/toolchain drift invalidates hashes broadly

sccache's cache key for a .cpp file is derived from its preprocessed content, which pulls in every transitively-included header. In a Boost-heavy codebase almost every translation unit includes at least one vcpkg-provided header. If the installed vcpkg package set differs even slightly between when the cache was last populated and a given run, effectively every TU that includes the affected headers gets a new hash simultaneously — which looks exactly like "near-total miss despite touching only ~20 files."

The user's sensible prior: the vcpkg submodule (port recipes/build baseline) should not move mid-sprint — it's only supposed to bump once, at end-of-sprint. Checked and confirmed true: 

$ git submodule status vcpkg
 44819aa2a6c10e56065e2b0330e7d6c89d1d2574 vcpkg (2026.06.01-163-g44819aa2a6)

$ git log -3 --format="%h %ai %s" -- vcpkg
0f96a1e30 2026-06-14 09:41:55 +0100 [vcpkg] Update to 2026.06.01-163-g44819aa2a6
fd91ce1b2 2026-06-12 13:03:56 +0100 [build] Update vcpkg submodule to latest master (b216ddff25)
9f36c3118 2026-06-07 14:54:20 +0100 [compass] Add timeline command: generate and show

The submodule pointer has been stable since 2026-06-14 — ~19 days before this PR's build, and unchanged for the whole of sprint 22 so far. So the submodule itself is not the cause.

But vcpkg.json (the manifest — dependency list + version-string, separate from the submodule pointer) did change twice this week, both times inside this very session's work: 

$ git log -5 --format="%h %ai %s" -- vcpkg.json
ec78e347f 2026-07-02 14:10:16 +0100 [build] Bump version to 0.0.22
dab2ec522 2026-07-02 03:24:26 +0100 Fix: add missing boost-circular-buffer dependency
9f3cfadd9 2026-06-12 13:21:49 +0100 [build] Bump version to 0.0.21
  • dab2ec522 added a real new port (boost-circular-buffer) to the dependency list — a genuine vcpkg-install-cache-relevant change (vcpkg-cache is keyed on hashFiles('vcpkg.json'), with a restore-keys prefix fallback so it should still partially reuse the prior cache, needing to build/cache just the one new port).
  • ec78e347f (the routine sprint-22 version bump, done as part of this same session's work) only changed the version-string field — no dependency change at all — but still changes the file's content hash, producing a new exact-match vcpkg-cache key on every version bump, purely from a metadata field.

Both commits landed on 2026-07-02, the day before PR #1410's canary run (2026-07-03 08:46). This is concrete, provable evidence that vcpkg.json — unlike the submodule — genuinely does change within a sprint, and did so twice in the 24h before the slow build. What isn't yet proven is the causal link to the 1.36% sccache hit rate specifically (vcpkg-cache and sccache are two separate Actions caches; a vcpkg-cache miss/rebuild for one new port doesn't automatically explain ~2400 sccache misses across the whole codebase). That link needs to be established with a live comparison, not log archaeology after the fact — see Tasks.

Also noted, not yet actioned

  • vcpkg install itself took ~20 minutes (08:51:40=→=09:12:18) inside the Run CTest workflow step, even with the vcpkg-cache (package binary cache) restoring ~1.9 GB successfully — worth understanding once the sccache question is settled, may be a related or separate cost.
  • The sccache-linux-gcc-debug-ninja and cmake-build-linux-gcc-debug-ninja caches both use a single static key (no content hash, no branch scoping) with an explicit delete-then-save step (gh api --method DELETE immediately before actions/cache/save) — this is a deliberate "always refresh" pattern, not neglect, but means whichever job's Delete=+=Save pair runs last "wins" when multiple canary jobs finish close together; worth confirming this doesn't cause its own thrashing independent of the vcpkg.json question.

Status

Field Value
State DONE
Parent sprint Sprint 22
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-07-03

Acceptance

  • Root cause of the near-zero sccache hit rate is identified with direct evidence (not inference) — e.g. a diff of sccache's actual hash inputs (SCCACHE_LOG=debug) between two back-to-back canary runs, showing exactly which input changed for a file that should have hit.
  • Either a fix lands (workflow/cache config change) and a subsequent canary run shows a meaningfully improved hit rate for an unrelated-file-only PR, or the story concludes with a documented reason why no fix is warranted right now (e.g. "confirmed vcpkg.json churn is the cause and it's expected to settle once sprint-22's remaining vcpkg-touching work lands").
  • No changes made to the vcpkg submodule pointer as part of this story — that only moves at end-of-sprint per existing convention.

Tasks

Task State Start End Description
Prove/disprove sccache cache-invalidation cause with live evidence DONE 2026-07-03 2026-07-03 Landed quota cleanup, max-size bump, and SCCACHE_LOG=debug telemetry; the access-scoping/quota finding is real but doesn't explain this run's full magnitude — analysis of the live experiment split into a follow-up task.
Analyze sccache zero-source-diff experiment results DONE 2026-07-03 2026-07-03 Decisive result obtained without needing the planned rerun: 0.00% hit rate, cache not found at all — evicted by cross-platform quota contention on GitHub's fixed 10 GB per-repo cache limit. Reverted the counterproductive max-size bump and removed the unhelpful debug instrumentation.
Make canary read-only for sccache/cmake-build caches DONE   2026-07-04 Confirmed root cause via GitHub's cache API: canary-linux.yml's Delete+Save steps mean every PR branch writes its own full-size scoped cache copy under the same key text (GitHub Actions caches are scoped per-ref even with identical keys). Found two live sccache-linux-gcc-debug-ninja entries (~2.1GB each) belonging to two different open PRs, and zero entries scoped to main — canary only runs on pull_request, so main never populates a baseline. With several PRs open concurrently, redundant per-branch copies fill the shared 10GB quota and starve everyone. Removed canary's Delete/Save steps for both caches, leaving only the existing Restore steps; continuous-linux.yml already reads+writes and will build the shared baseline canary reads from.

Decisions

Final finding (2026-07-03, same day, live experiment): PR #1417 (a trigger PR touching only a workflow comment — zero C++ diff from what had just been compiled minutes earlier) ran canary and got a 0.00% hit rate (0/2435), even worse than PR #1410's 1.36%. Crucially, the restore step didn't even find a cache to try: 

Cache not found for input keys: sccache-linux-gcc-debug-ninja

Checking gh api repos/.../actions/caches at that moment explained why: the main-scoped sccache-linux-gcc-debug-ninja entry had been evicted entirely. The repo's cache list showed 12 other entries competing for the same fixed 10 GB total repository quota: sccache-windows-clang-debug-ninja (1083MB), sccache-macos-clang-debug-ninja (1712MB), sccache-windows-clang-release-ninja (802MB), sccache-macos-clang-release-ninja (602MB), plus their vcpkg-cache and install-qt-action caches — all recently touched by continuous-*.yml runs on other platforms, all sharing the same 10 GB pool as the Linux/gcc canary cache. GitHub's LRU eviction had simply dropped the Linux-gcc entry to make room for the others.

This is the real mechanism, and it also explains PR #1410's partial (1.36%, not 0%) result: whether the Linux-gcc-debug cache survives at all, at any given moment, depends on which OS/compiler caches were touched most recently across the entire repo, not on anything specific to a given PR's diff. The earlier max-size 2000M→3000M bump (from the first, premature pass at this story) was counterproductive — it makes each cache entry larger, consuming more of the shared 10 GB pool and accelerating eviction of the other platforms' caches (and vice versa). Reverted it back to 2000M in both canary-linux.yml and continuous-linux.yml. Also removed the SCCACHE_LOG=debug instrumentation — it turned out to add no value (pure per-invocation client boilerplate, no useful hash/lookup detail at that log scope) and the decisive evidence came from the plain actions/cache/restore log line and the GitHub cache API directly, not from sccache's own logging.

What this means going forward: GitHub's 10 GB per-repo cache quota is a hard, non-purchasable platform limit (same across all plan tiers) — it cannot be raised. ORE Studio currently runs 6+ distinct build variants (linux-gcc × {debug,release}, linux-clang × {debug,release}, windows-clang × {debug,release}, macos-clang × {debug,release}, roughly) all sharing that one pool alongside vcpkg/Qt caches, and the combined working set is evidently larger than 10 GB — hence constant thrashing. The durable fix is almost certainly to move sccache off GitHub's built-in actions/cache onto an external backend sccache natively supports (S3/GCS/Azure Blob, or a self-hosted Redis/webdav endpoint) — bounded only by storage you provision, not shared across unrelated platforms, and not subject to GitHub's 7-day-unused eviction. That is a bigger infrastructure change (needs a bucket/credentials, IAM setup, workflow changes across every platform's workflow file) than fits in this story's scope — recorded here as the follow-up recommendation rather than attempted now.

Correction (2026-07-03, same day): the entry below was written after PR #1410's canary run and initially treated as settling the story (task/story briefly marked DONE). Checked the numbers more carefully after being challenged on them: the main-scoped cache that run restored was only ~20 minutes stale (built by continuous-linux.yml at 08:45, canary started 08:46), and the actual file diff between that cache's basis commit and PR #1410's build was only 32 files (git diff --name-only <continuous's commit> <PR's commit>). 32 changed files cannot plausibly explain 2400/2433 (98.6%) cache misses even with generous header-cascade effects. So the access-scoping/quota explanation below is a real, worth-fixing architectural gap (it explains why the cache never improves across days), but it is not the proximate cause of this specific run's near-total miss rate. That cause is still unknown. Reopened; running a live zero-source-diff experiment (see Plan) instead of relying on inference.

Root cause: GitHub Actions cache access control, not vcpkg.json churn directly. A PR job can only restore from its base branch's (main) cache, never save to it — by design, to stop a PR (and especially a fork PR) from poisoning the shared cache. Since canary-linux.yml only runs on pull_request (no push-to-main trigger), the main-scoped sccache-linux-gcc-debug-ninja cache is only ever refreshed by continuous-linux.yml (which shares the identical cache-key scheme via its linux/gcc/debug matrix leg). Every PR's own incremental compiles are thrown away the moment its PR-ref-scoped cache becomes unreachable (merge/close) — PRs never compound on each other, only on whatever the last continuous run produced. This was compounded by the repo sitting at 9.79/10 GB of GitHub's per-repo cache quota, with dead refs/pull/1410/merge-scoped entries (already-merged PR) crowding out room to grow.

vcpkg.json churn (boost-circular-buffer addition + version bump, both 2026-07-02) remains a plausible secondary contributor but was not proven as the primary cause — the access-scoping mechanism above is sufficient on its own to explain a near-zero hit rate regardless of vcpkg.json. Confirmed per the user's expectation: the vcpkg submodule pointer has not moved this sprint and was not touched by this story.

Scope of the fix: a quick, low-risk pass — freed ~2.7 GB by deleting the orphaned PR-1410 cache entries, and raised sccache max-size 2000M → 3000M for the shared key in both workflows that write it. A full live SCCACHE_LOG=debug before/after hash-input comparison was scoped but deferred — it needs two full ~3h canary runs and isn't proportionate to "quick"; the mechanism above is well-evidenced via the GitHub cache API directly, not inferred from job-log archaeology.

Closing fix: canary-linux.yml's own Delete+Save steps were a direct contributor to the quota contention diagnosed above — every PR branch wrote its own full-size scoped copy under the same key text (GitHub Actions caches are scoped per-ref even when the key string is identical), so N concurrently open PRs meant N redundant ~2GB+ copies competing for the same fixed 10GB quota. Confirmed live via the cache API: two sccache-linux-gcc-debug-ninja entries existed simultaneously, one per open PR, and zero existed scoped to main (canary only triggers on pull_request, so main never got a chance to seed a shared baseline). Made canary read-only (removed its Delete/ Save steps for both the sccache and cmake-build caches), leaving continuous-linux.yml — which already reads and writes under the identical key and runs on main via its cron schedule — as the sole writer. GitHub Actions caches created on the default branch are readable from any other branch, so canary's restore step will now pick up a single shared, continuously-warmed baseline instead of every PR cold-building and hoarding its own copy.

Out of scope

  • Moving the vcpkg submodule pointer (only happens once, at end-of-sprint, per existing convention).
  • A live SCCACHE_LOG=debug hash-input diff proving the exact invalidation trigger file-by-file — turned out unnecessary once the cache-quota mechanism was found directly via the GitHub API; see Decisions.
  • Migrating sccache to an external cache backend (S3/GCS/Azure/ self-hosted) to escape GitHub's 10 GB shared-quota contention — this is the actual durable fix the evidence points to, but is a real infra project (bucket/credentials/IAM, workflow changes across every platform). Captured as a backlog item (doc/agile/product_backlog/inbox/sccache_external_cache_backend.org) rather than attempted in this story.

Emacs 29.3 (Org mode 9.6.15)