Story: Migrate shared ccache/build storage to btrfs for reflink-safe hardlink-free caching

KargaFormacao/Karga is a full .NET source tree (Karga.sln, Karga.Core, Karga.WebApi, Karga.Blazor, Karga.Mobile, Karga.Cli) carrying its own .git. It was missed by the first repo sweep because that used find -maxdepth 4 and this sits at depth 5 – see D6.

Its remote is HTTPS (https://github.com/Moimba/Karga.git), not SSH, so it does not authenticate via the agent configured in Step 0b. It held six uncommitted files of real work (TallySheet3 domain, entity, repository and endpoint classes). Resolved 2026-07-28:

• Committed and pushed to wip/kargaformacao-tallysheet3 on git@github.com:Moimba/Karga.git (pushed over SSH via an explicit push URL; the repo's HTTPS remote config was left unchanged). Verified on the remote at e66226b44.
• The whole 395M tree is also copied to /home/marco/Development/KargaFormacao, byte-identical.

It points at /mnt/development/Development/Moimba. Anything copied "into home's Moimba" lands back on the partition being destroyed. Backup copies must go to a sibling path – /home/marco/Development/KargaFormacao, not /home/marco/Development/Moimba/KargaFormacao.

The symlink will dangle after the reformat until Moimba is recreated.

Today the SSD is a single ext4 partition plus a 16G swapfile inside it (pri=100, 3.1G in use). btrfs swapfiles carry real constraints: they must be nodatacow, cannot be compressed, and forbid snapshots of the containing subvolume. A dedicated swap partition avoids all of it, so the SSD is repartitioned:

8G rather than the current 16G: the host has 31G RAM and is using only 3.1G of swap, and /dev/sdb3 already exists as a 64G swap partition sitting unused at pri-1=. Tiering fast SSD swap at pri=100 over slow overflow swap at pri-1= keeps the current behaviour and returns 8G to the SSD. (Note /dev/sdb3 is on the rotational disk – it is overflow insurance, not a working tier.)

Subvolumes are free, cost nothing when unused, and let PostgreSQL get different mount semantics from everything else. Created at the top level of the new filesystem:

PostgreSQL's 8K random rewrites are the one workload that genuinely fragments badly under COW. chattr +C must be applied to the directory while it is empty, before initdb – applying it afterwards does nothing for existing files. This is the single most common btrfs + database mistake and the story previously listed it as out of scope.

Confirmed topology: the fleet is already git worktree-based – 8 worktrees hang off ores_dev_prime_origin/.git, sharing one 1.8G object store. That is good, but it makes prime_origin a load-bearing special case: deleting or corrupting it breaks the other 8. The clean form is a bare repository with 9 equal worktrees, none privileged.

btrfs kernel support is present (btrfs.ko ships with kernel 7.1.4+deb14-amd64) but btrfs-progs was not installed – mkfs.btrfs did not exist on this host. Install the toolchain first; discovering this after umount would be painful.

Note two package names that are not what you would guess: compsize is shipped as btrfs-compsize, and sgdisk (needed by Step 3, i.e. after the umount) comes from gdisk, which was also absent.

sudo apt update
sudo apt install btrfs-progs duperemove btrfs-compsize gdisk
mkfs.btrfs --version   # must succeed before proceeding
sgdisk --version       # ditto -- Step 3 cannot run without it

Installed and verified 2026-07-28: btrfs-progs v7.0, duperemove 0.15.2, btrfs-compsize 1.5, gdisk/sgdisk 1.0.10. modprobe btrfs succeeds and btrfs appears in /proc/filesystems.

/dev/sda already carries a GPT label (disk identifier D5A5F2FA-95B5-48DE-BF5A-DA950B1B0654, single 223.6G partition), so Step 3's sgdisk commands need no MBR-to-GPT conversion.

apt update reports pre-existing errors from two third-party repos (timescaledb-for-Ubuntu and a Microsoft mssql key). Unrelated to this work and harmless here – everything needed comes from Debian main.

• Confirm no other user is on neumann; no build or service may run during Steps 1-3.

• Note the current state for rollback reference (captured 2026-07-28 to /home/marco/, which lives on /dev/sdb2 and is untouched by this migration):

  lsblk -f > ~/pre-migration-lsblk.txt; blkid >> ~/pre-migration-lsblk.txt
  swapon --show > ~/swap.pre-migration.txt
  df -hT /mnt/development >> ~/swap.pre-migration.txt
  cp /etc/fstab ~/fstab.pre-migration
  cp /home/marco/.config/ccache/ccache.conf ~/ccache.conf.pre-migration
  # PostgreSQL cluster config -- pg_dropcluster in Step 6 deletes it:
  sudo cp -a /etc/postgresql/18/main ~/postgresql-18-main.conf.pre-migration
  

  The PostgreSQL config copy is listed again under Step 2; doing it here is equivalent and gets it out of the way.

• Keep an off-SSD copy of this story, since the disk holding it is the one being destroyed:

  cp story.org ~/migrate-ccache-to-btrfs-reflink-story.org   # survives reboot
  cp story.org /tmp/                                          # tmpfs: does NOT
  

  /tmp on this host is tmpfs (15.6G, RAM-backed), so a copy there is lost at the reboot in Step 4. $HOME is on /dev/sdb2, untouched by this migration.

Step 6 re-clones from GitHub over SSH, so agent access must survive the reboot. It previously did not: SSH_AUTH_SOCK was supplied by sourcing ~/crap.sh, a saved env snapshot pointing at a hand-started ssh-agent (PID 1547, parented to init), which went stale on every boot. Four agents were running concurrently as a result.

No new unit was needed – openssh-client already ships ssh-agent.service / ssh-agent.socket in /usr/lib/systemd/user/, already enabled, listening on %t/openssh_agent. The only gap was that nothing exported SSH_AUTH_SOCK to shells. Now configured:

~/.bashrc alone is insufficient: it returns early for non-interactive shells, hence the duplicate block in ~/.profile. ~/.ssh/config gained a github.com block (IdentityFile ~/.ssh/id_rsa, IdentitiesOnly yes) and a trailing Host * default of AddKeysToAgent yes, so the passphrase is entered once per boot rather than once per connection. loginctl show-user marco -p Linger already reports Linger=yes, so the agent starts without an interactive login.

~/crap.sh is now unused and can be deleted.

After the Step 4 reboot, confirm before relying on it:

systemctl --user status ssh-agent.socket
echo "$SSH_AUTH_SOCK"          # expect /run/user/1000/openssh_agent
ssh-add -l                     # expect the RSA key, may prompt for passphrase
git ls-remote git@github.com:OreStudio/OreStudio.git >/dev/null && echo OK

No bulk backup, and no PostgreSQL dump – the cluster is recreated from scratch (D5). Build output, vcpkg trees, NATS state, the ccache store and all database content are disposable by design. Two things are not:

1. Uncommitted or unpushed git work. Push it to the remote – not to local disk.

   Do not enumerate repositories by hand. The original version of this step looped over nine hardcoded worktree names and would have missed 17 of the 26 repositories actually present (D6). Discover them:

   for p in $(find /mnt/development -name .git -not -path "*/vcpkg*" \
                   -not -path "*/node_modules/*" | sed 's|/\.git$||' | sort); do
       d=$(git -C "$p" status --porcelain | wc -l)
       u=$(git -C "$p" log --branches --not --remotes --oneline | wc -l)
       s=$(git -C "$p" stash list | wc -l)
       [ "$d$u$s" != "000" ] && printf '%-60s dirty=%-4s unpushed=%-4s stash=%s\n' \
           "${p#/mnt/development/Development/}" "$d" "$u" "$s"
   done
   

   Note -maxdepth is deliberately absent: KargaFormacao/Karga sits at depth 5 and Claude Code agent worktrees under <worktree>/.claude/worktrees/ sit deeper still. Both hold real work. vcpkg and node_modules are pruned because they contain vendored third-party repos, not our work. Gate: this sweep must come back clean, or every remaining item must be explicitly accepted as discarded, before Step 3 runs.

   State captured 2026-07-28, before any destructive step – this is the assignment Step 6 restores:

   Worktree	Branch	Dirty	Unpushed
   ores_dev_swift_curie	feature/debug-start-all-never-resumes-in-container	0	--
   ores_dev_prime_origin	feature/migrate-to-oresmd-delete-legacy-schemes	0	--
   ores_dev_brave_hopper	feature/simulate-holding-group-provisioning-via-internal-impersonation	0	--
   ores_dev_bright_faraday	feature/extend-synthetic-binding-dialog-to-ir-curves	0	--
   ores_dev_jolly_knuth	feature/add-entity-filter-flag-to-regenerate	0	--
   ores_dev_solid_dirac	feature/write-up-wire-format-architecture	0	--
   ores_dev_merry_newton	feature/merge-asset-class-folders-into-one-theme-unit	1	--
   ores_dev_eager_maxwell	feature/badge-source-self-annotation	0	--
   ores_dev_clever_dijkstra	feature/codegen-qt-parent-scoped-list	0	--

   All outstanding work was found and resolved on 2026-07-28. What the full sweep turned up, well beyond the nine worktrees:

   Repo	Was	Resolution
   Hedgr.remote	6 dirty, 4 unpushed	committed; branch feature/initial_infrastructure pushed (it had no upstream at all)
   Moimba/Baobab.local1	12 dirty, 1 unpushed	committed and pushed; out/ + tmp/ gitignored
   Moimba/Karga.local1	5 dirty	committed and pushed
   Moimba/KargaFormacao/Karga	6 dirty	pushed to wip/kargaformacao-tallysheet3; also copied to $HOME
   Moimba/Corimba.remote, Corimba.plan30	1 dirty each	.claude/settings.json committed and pushed
   Papers	4 dirty	committed and pushed; also copied to $HOME
   prime_origin/.claude/worktrees/agent-af474...	2 dirty (283 lines)	committed; pushed to worktree-agent-af474c160d620e68c
   OreStudio ×9	1 unpushed (f8c7241a5)	pushed. Reported from every worktree because they share one object store – one commit, not nine
   Engine.remote	1 dirty	deliberately left – see below

   Two deliberate non-actions:

   • Engine.remote is a fork (origin OreStudio/Engine, upstream OpenSourceRisk/Engine) whose only change is incidental submodule checkout drift on ORE-SWIG/QuantLib-SWIG, on a master that is 1324 commits behind origin. No work was done in it. It is re-cloned in Step 6, which also brings it current.
   • Stashes (3 shared across the OreStudio worktrees, 1 in each Baobab clone) are accepted as discarded, by explicit decision.

2. Hedgr.remote / Engine.remote (4.8G / 2.2G): verify clean against upstream, then re-clone rather than copy.

   for r in Hedgr.remote Engine.remote; do
       echo "=== $r ==="; cd /mnt/development/Development/OreStudio/$r
       git status --short; git log --branches --not --remotes --oneline
   done
   

The PostgreSQL cluster configuration at /etc/postgresql/18/main lives on /, not on the SSD, so it survives the reformat untouched – but Step 6 drops and recreates the cluster, which deletes it. Copy it aside first so any local tuning can be diffed back in:

sudo cp -a /etc/postgresql/18/main ~/postgresql-18-main.conf.pre-migration

mkdir -p /mnt/development/.cache/ccache
ccache --set-config=cache_dir=/mnt/development/.cache/ccache
ccache --set-config=hard_link=false     # THE BUG: currently true
ccache --set-config=file_clone=true
# Entries are now stored uncompressed (compression is force-disabled by
# file_clone), so the same hit rate needs a much larger nominal budget.
# The filesystem compresses it back down on disk.
ccache --set-config=max_size=150G
ccache --zero-stats
ccache --show-config | grep -E 'hard_link|file_clone|max_size|cache_dir'

Keep direct_mode=true and hash_dir=false exactly as they are – hash_dir=false is what makes cross-worktree sharing possible at all, and changing it would silently defeat this whole story. Both confirmed still set after the reconfiguration.

Bare repo first, so no worktree is privileged:

Layout change: there is no Development/ level any more. It was a redundant directory inside a mount already called /mnt/development. Projects sit directly under the mount point:

/mnt/development/OreStudio/...
/mnt/development/Moimba/...
/mnt/development/.cache/ccache
/mnt/development/postgres        (the @postgres subvolume)

Every old path of the form /mnt/development/Development/OreStudio/<x> becomes /mnt/development/OreStudio/<x>.

mkdir -p /mnt/development/OreStudio
cd /mnt/development/OreStudio
git clone --bare git@github.com:OreStudio/OreStudio.git ores.git
cd ores.git
git config remote.origin.fetch '+refs/heads/*:refs/remotes/origin/*'
git fetch origin

# Then all nine as equal siblings, restoring the branch assignment from
# the table in Step 2:
git worktree add ../ores_dev_swift_curie     feature/debug-start-all-never-resumes-in-container
git worktree add ../ores_dev_prime_origin    feature/migrate-to-oresmd-delete-legacy-schemes
git worktree add ../ores_dev_brave_hopper    feature/simulate-holding-group-provisioning-via-internal-impersonation
git worktree add ../ores_dev_bright_faraday  feature/extend-synthetic-binding-dialog-to-ir-curves
git worktree add ../ores_dev_jolly_knuth     feature/add-entity-filter-flag-to-regenerate
git worktree add ../ores_dev_solid_dirac     feature/write-up-wire-format-architecture
git worktree add ../ores_dev_merry_newton    feature/merge-asset-class-folders-into-one-theme-unit
git worktree add ../ores_dev_eager_maxwell   feature/badge-source-self-annotation
git worktree add ../ores_dev_clever_dijkstra feature/codegen-qt-parent-scoped-list

Remote verified reachable 2026-07-28 via git ls-remote git@github.com:OreStudio/OreStudio.git. SSH auth is via the systemd-managed agent (see Step 0), not the old ~/crap.sh env snapshot.

PostgreSQL cluster once, created fresh into the nodatacow subvolume (see devops-provision-environment for the scripted form). The Debian cluster definition at /etc/postgresql/18/main survives the reformat but points at a data directory that no longer exists, so drop and recreate it rather than trying to repair it:

# The old cluster is now a dangling definition -- data dir is gone.
sudo pg_dropcluster 18 main --stop

# Recreate with the data directory on the nodatacow subvolume. The dir
# must not already contain data; pg_createcluster runs initdb into it,
# and every file it creates inherits the +C flag set in Step 4.
sudo pg_createcluster 18 main -d /mnt/development/postgres/18/main
sudo systemctl start postgresql@18-main
pg_lsclusters                     # expect: 18 main 5432 online

# Confirm nodatacow was actually inherited by the new data directory:
sudo lsattr -d /mnt/development/postgres/18/main   # expect ---------------C------

Scope: Corimba only. Karga and Baobab are not recreated on the SSD. All of their work was committed and pushed during Step 2, so nothing is lost – they can be cloned back at any time if needed. This also returns ~5.8G (Karga.local1 3.5G, Baobab.local1 2.1G, plus the two small clones) to the disk.

Corimba adopts both OreStudio conventions (D7): a bare repo with equal worktrees (D3), and the <project>_dev_<adjective>_<scientist> naming scheme. The old Corimba.remote / Corimba.plan3x names are retired – Corimba.remote was a clone that also hosted the other worktrees' object store, exactly the load-bearing arrangement D3 exists to remove.

Names are chosen not to collide with the nine OreStudio ones. Corimba.plan31 was detached and clean, so its slot is given to the branch the old Corimba.remote clone was on – no branch is lost.

mkdir -p /mnt/development/Moimba
cd /mnt/development/Moimba

git clone --bare git@github.com:Moimba/Corimba.git corimba.git
git -C corimba.git config remote.origin.fetch '+refs/heads/*:refs/remotes/origin/*'
git -C corimba.git fetch origin

git -C corimba.git worktree add ../corimba_dev_keen_lovelace feature/tidy_up_employee_dataset
git -C corimba.git worktree add ../corimba_dev_bold_turing   main
git -C corimba.git worktree add ../corimba_dev_wise_shannon  feature/cycle-phases-architecture

KargaFormacao is not restored to the SSD, consistent with dropping Karga. It stays at /home/marco/Development/KargaFormacao (395M, byte-identical copy taken 2026-07-28), and its committed work is on Moimba/Karga at wip/kargaformacao-tallysheet3. It is the one tree not reproducible from a remote alone, so do not delete the $HOME copy – it is now the only full copy of that layout.

Finally, repoint the /home/marco/Development/Moimba symlink. It targets /mnt/development/Development/Moimba, a path that no longer exists under the new layout, so it dangles from the moment of the reformat and would still dangle even after Moimba is recreated:

ln -sfn /mnt/development/Moimba /home/marco/Development/Moimba
ls -l /home/marco/Development/Moimba && ls /home/marco/Development/Moimba/

Reflinks only arise where something clones a file. The ~1.2G vcpkg_installed tree that each worktree builds independently is near-identical across all nine (~11G for ~1.2G of content) and nothing clones it. An offline dedup pass converts those into shared extents:

# Whole mount, not just OreStudio: Moimba's five worktrees share vendored
# NuGet/obj trees with each other in the same way.
sudo duperemove -rdh --hashfile=/var/tmp/dev-dedup.hash /mnt/development
sudo compsize /mnt/development

Re-run after any large rebuild wave. compsize reports both the compression ratio actually achieved and the extent sharing. It must be run as root – unprivileged it prints nothing rather than failing.

Exclude /mnt/development/postgres from any future dedup pass if one is scripted: deduplicating a NOCOW database file reintroduces shared extents, which silently converts writes back into copy-on-write and undoes Step 4. The run below was scoped to OreStudio and Moimba explicitly for this reason.

Correctness – the bug this story exists to fix:

• ccache --show-config reports hard_link = false, file_clone = true.
• A build in two different worktrees of the same commit produces a ccache hit in the second one (confirms the shared cache still works) without the second worktree's object file being a hard link (stat -c '%h' <file>.o reports link count 1, not >1).
• Full compass build rat green in at least two worktrees run concurrently, deliberately racing the same TU, with no crash – the regression test for the bug this story fixes.

Space – the constraint that drove the filesystem choice:

sudo compsize /mnt/development/Development/OreStudio   # ratio + shared extents
sudo btrfs filesystem usage /mnt/development           # real free space, not df
df -h /mnt/development                                 # for contrast; trust the above

• compsize reports a compression ratio materially below 1.0 (target: roughly 0.35-0.5 across the build trees) – if it reports ~1.0, the compress=zstd:3 mount option did not take, and the entire rationale for choosing btrfs over XFS has silently evaporated. This is the check that matters most.
• btrfs filesystem usage shows unallocated space remaining. Chunk exhaustion, not df, is what causes btrfs ENOSPC.
• Total footprint after all nine worktrees are built is below the 173G the ext4 layout consumed, despite ccache now storing entries uncompressed.

Configuration:

• lsattr -d /mnt/development/postgres shows the C (nodatacow) flag.
• swapon --show lists /dev/sda2 at pri=100 and /dev/sdb3 at pri-1=; no swapfile on the btrfs filesystem.
• A reboot mounts everything from /etc/fstab with no manual intervention.

Every acceptance claim verified on real builds, not inferred.

Fleet rebuilt after merging the ccache-default fix and rebasing every worktree onto main. All nine: exit 0, zero errors.

Eight full rebuilds at 99.6-99.9% in ~4 minutes each, against ~40 minutes cold. The whole fleet rebuilt in about the time one cold build takes. Note the contrast with the 72.5% measured earlier across divergent branches: once every worktree shares a common main base, reuse is essentially total.

Builds were run strictly one at a time. Concurrent builds contend on the vcpkg lock – the failure mode that stalled Step 6 – and serial runs keep per-worktree hit rates interpretable.

Not required for acceptance, but this is what btrfs costs versus XFS and it should be written down rather than rediscovered:

• If btrfs filesystem usage shows unallocated space approaching zero while df still claims free space, run sudo btrfs balance start -dusage=50 /mnt/development.
• Re-run duperemove (Step 7) after large rebuild waves.
• Never enable snapshots on @development without revisiting the space budget – snapshots are the one thing that turns COW into standing overhead, which is precisely the failure mode this design avoids.