Task: Per-environment systemd resource limits (Claude slices, services, builds)

Table of Contents

This page documents a task in the Systemd resource management and per-environment isolation story. It captures the goal, current status, acceptance, and any notes or results.

Goal

Make compass the single wrapper for process lifecycle across the fleet, so a runaway environment's memory usage can only kill that environment, never Emacs, the host, or another environment's fleet. See the design plan for the full root-cause analysis (2026-07-31 machine lockup) and workstream breakdown (WS-1 through WS-7, WS-6 deferred).

Landed as a sequence of small, independently reviewed PRs following the plan's own sequencing, not one large change – each workstream touches a different file/subsystem and has its own verification.

Status

Field Value
State DONE
Parent story Systemd resource management and per-environment isolation
Now Nothing.
Waiting on Someone with root, for WS-5's root half – left as a follow-up, not blocking this task's close.
Next Nothing.
Last touched 2026-08-05

Acceptance

(From the plan's own * Acceptance, carried here verbatim as this task's bar for DONE.)

  • compass claude launches into app-claude-<env>.slice, and systemctl --user show on that slice reports a non-infinite MemoryMax.
  • compass services start results in service processes whose /proc/<pid>/cgroup is under the environment's slice, regardless of whether it was invoked from Emacs, a Claude session, or a bare shell.
  • compass services status shows exactly the current environment's fleet, including when nothing is running.
  • Deliberately exhausting memory in one environment kills only that environment; Emacs and other fleets survive. Tested explicitly, with MemoryMax temporarily lowered so the test is quick and safe.
  • compass build --status reports two slots, and a running build appears under app-build-<env>.slice in systemd-cgls.
  • No compass subcommand requires root for normal operation.

Plan

Sequencing per the design plan's * Sequencing section:

  1. WS-7 change 1 – drop build lock slot c (two slots instead of three). Small, immediately reduces peak memory.
  2. WS-1 – per-environment Claude slices with limits.
  3. WS-5 (unprivileged half) – emacs.service drop-in.
  4. WS-4 – per-environment status/tree/top views.
  5. WS-7 remainder – builds into their own memory-capped slice.
  6. WS-3compass systemd deploy. Discovered already implemented when WS-5's deploy logic was folded into it; no separate PR needed (see the WS-5 note below).
  7. WS-2 – converge compass services onto systemd (the large one, staged behind --legacy). Discovered already implemented during WS-4 (compass_services.py has been fully systemctl-backed, no PID-file path, since Sprint 24's decommission-controller-service task, PR #1806 – predates this story); no separate PR needed.
  8. WS-5 (root half) – once run with root, verify propagation.
  9. WS-6 – deferred; only if still needed after the above.

Each numbered step lands as its own PR against this task's branch lineage (a fresh branch per step, closed/reopened as needed), with its own local verification and review round, per this repo's normal task-to-merged-PR lifecycle.

Notes

WS-7 change 1: drop build lock slot 'c'

BUILD_LOCK_SLOTS in projects/ores.compass/src/compass.py reduced from three slots (a-j3, b=-j2, c=-j2=, 7 cores in flight) to two (a-j3, b=-j2=, 5 cores in flight), matching the plan's DD/WS-7 change 1 exactly. cmd_build's docstring and doc/recipes/cmake/how_do_i_build_the_system.org already described "two slots"/"two environments" – they were already correct, so no further doc drift to fix here (the plan's claim that the docstring was stale referred to language now already updated). Verified with compass build --status: shows exactly two slots (a, b), no c.

WS-1: per-environment Claude slices with limits

compass_claude.py changes, matching the plan's three changes exactly:

  1. _SLICE_NAME (module constant) replaced with _slice_name(env_name), nesting each session in app-claude-<env>.slice instead of the flat app-claude.slice. systemd creates the per-environment slice implicitly from the --slice=app-claude-<env>.slice passed to systemd-run; no new unit file needed.
  2. New checked-in drop-in source, projects/ores.compass/src/systemd/app-claude-.slice.d/50-limits.conf (dash-truncated template — applies to every app-claude-<env>.slice with one file), carrying MemoryHigh=6G=/=MemoryMax=9G=/ =MemorySwapMax=2G=/=CPUWeight=100 verbatim from the plan.
  3. _ensure_slice_deployed() generalised from a single (source, dest) pair to a manifest list, so the slice unit and the limits drop-in both sync (independently — only copies+reloads what actually changed) in one pass.

Verified directly (not via a live compass claude launch, to avoid disrupting this session's own running scope): called compass_claude._ensure_slice_deployed() directly, confirmed both files land correctly under ~/.config/systemd/user/, then started a throwaway systemd-run --user --scope --slice=app-claude-brave_hopper.slice /bin/true and confirmed via systemctl --user show that MemoryMax=9663676416 (9G), MemoryHigh=6442450944 (6G), MemorySwapMax=2147483648 (2G), CPUWeight=100 all apply — matching the plan's own worked example. Confirmed app-claude.slice itself is unaffected (MemoryMax=infinity, accounting only), and that pre-existing, already-running Claude sessions (launched before this change) correctly remain under the old flat slice — no live session was disrupted by testing this.

WS-5 (unprivileged half): protect Emacs

New checked-in drop-in, projects/ores.compass/src/systemd/emacs.service.d/50-protect.conf, carrying MemoryMin=1G=/=MemoryLow=2G=/=CPUWeight=10000 verbatim from the plan. Unlike WS-1's drop-in, this one isn't per-environment (there's one Emacs daemon per host, not per checkout), so it's a plain static file rather than a dash-truncated template.

Deployment folded into the existing compass systemd deploy (systemd_generate.py's cmd_deploy) rather than a new command: a STATIC_DROPINS manifest of (source, dest) pairs, synced with the same copy-if-changed logic already used for the generated per- environment units, sharing the same added=/=updated=/=removed report and the same single daemon-reload gate. This also confirms, incidentally, that WS-3 was already done before this task started – compass systemd deploy already installs generated units and runs daemon-reload only when something changed (the plan describes this as still-to-build, but it was already implemented; no separate WS-3 PR is needed).

Verified: compass systemd deploy reported Added: 1 with + emacs.service.d/50-protect.conf; confirmed the file landed correctly at ~/.config/systemd/user/emacs.service.d/50-protect.conf with the right content; systemctl --user show emacs.service -p MemoryMin -p MemoryLow -p CPUWeight reports CPUWeight=10000, MemoryMin=1073741824 (1G), MemoryLow=2147483648 (2G) – CPUWeight is proportional and unprivileged so it's already fully effective; MemoryMin=/=MemoryLow are set at this level but per the plan's own caveat are clamped by ancestor cgroups until the WS-5 root half (/etc/systemd/system/user-1000.slice.d/50-marco.conf, requires root, not part of this PR) also lands – documented in the drop-in's own header so this isn't mistaken for a completed protection. Re-ran compass systemd deploy and confirmed it reports Nothing changed; skipped daemon-reload. (idempotent).

WS-4: per-environment tree/top views

Discovered while starting this step that WS-2 was also already done: compass_services.py's module docstring and cmd_status=/ =cmd_start=/=cmd_stop are already fully systemctl --user-backed – no PID-file path, no --legacy flag – since Sprint 24's decommission-controller-service task (PR #1806), which predates this story entirely. The plan's WS-2 table (PID-file "today" column) and WS-4's own status row (systemctl --user list-dependencies ores@<env>.target) describe a migration that had already happened by the time this task started; cmd_status actually does per-unit is-active plus readiness-log classification, which is finer-grained than a bare list-dependencies and was left as-is rather than reimplemented to match the plan's literal suggestion.

Added the two views the plan actually asks for that didn't exist yet, compass services tree and compass services top, in compass_services.py:

  • treesystemd-cgls across this environment's Claude session slice (app-claude-<env>.slice, from WS-1) plus every unit this environment's fleet aggregates (nats + every service, filtered to ones actually loaded via _unit_active_state() ! "missing"= – passing a nonexistent unit name aborts cgls entirely, confirmed empirically), one subtree per unit in a single invocation.
  • topsystemd-cgtop filtered to that same Claude slice. systemd-cgtop takes a real cgroupfs path, not a unit name (unlike cgls's --user-unit), so _claude_slice_path() builds the absolute unified-hierarchy path from os.getuid() rather than hardcoding the uid.

Real gap surfaced, not fabricated: fleet service units have no per-environment slice of their own today – only Claude sessions (WS-1) and, once WS-7's remainder lands, builds do. Confirmed live: systemctl --user show ores.iam.service-brave_hopper -p Slice reports app.slice, the flat default, not a per-environment child. So top's "filtered to the environment's slices" (plural, per the plan) is honestly only the Claude slice for now; tree's fleet section compensates by listing each service unit's own individual cgroup instead of one shared per-environment parent. Both subcommands forward unrecognised trailing flags verbatim (extra, nargs=REMAINDER) to the wrapped systemd-cgls=/=systemd-cgtop binary, matching the plan's "forward unrecognised arguments" requirement.

Verified: compass services tree printed one systemd-cgls subtree per loaded unit for this environment (nats-server plus all 23 service units, including the ores.compute.wrapper replicas), confirmed against the running fleet; compass services top -1 -b ran cleanly (exit 0) against the per-environment Claude slice path.

Correction from review round 1 (see * Review #3/#4): the initial verification's "empty" top output was wrongly attributed entirely to "no session predates WS-1 nesting" – review correctly identified a real path-construction bug (_claude_slice_path skipped the intermediate app-claude.slice segment systemd's dash-hierarchy actually requires) that would have made top point at a non-existent cgroup regardless of whether any session was running. Fixed, and re-verified live: top now reports a real, non-empty row for app-claude-brave_hopper.slice. tree had a related latent bug – it added the Claude slice to --user-unit unconditionally, with no existence check, unlike the fleet units it already guards the same way – which would abort the entire cgls call (fleet subtrees included) for any environment with no active Claude session. Fixed with a new _slice_is_loaded() check; verified directly against both a real loaded slice and a fabricated nonexistent one.

WS-7 remainder: memory-capped build slice

New checked-in drop-in, projects/ores.compass/src/systemd/app-build-.slice.d/50-limits.conf (same dash-truncated-template mechanism as WS-1's app-claude-.slice.d/). Values are this task's own estimate, not measured – the plan explicitly left build-slice numbers as an open question (same caveat WS-1's starting 6G/9G came with); documented in the drop-in's own header, sized for the heavier build-lock slot (a, -j3): MemoryHigh=8G=/=MemoryMax=12G=/=MemorySwapMax=3G=/ =CPUWeight=100. Revisit from real systemd-cgtop peaks once this has run for a while, per the plan's own suggestion.

In compass.py: _ensure_build_slice_deployed() (manifest-of-one, same copy-if-changed pattern as WS-1/WS-5, deliberately kept separate from compass_claude.py's manifest – build-lifecycle deployment belongs with the build code) and _build_slice_name(). cmd_build now wraps only the actual cmake --build step (not the configure step) in systemd-run --user --scope --slice=app-build-<env>.slice when a user systemd manager is available (_has_user_systemd(), same check compass_claude.py uses) and it isn't a dry run; --direct builds (Emacs scripts, no cmake) are unaffected, matching the plan's own scope (WS-7 is about cmake --build specifically).

Verified live end-to-end with a real (small, fast) build, compass build ores.utility.lib: confirmed app-build-brave_hopper.slice existed and was active while the build was running (systemctl --user list-units); confirmed via systemctl --user show that all four limits applied exactly (MemoryMax=12884901888 = 12G, MemoryHigh=8589934592 = 8G, MemorySwapMax=3221225472 = 3G, CPUWeight=100); build completed successfully (Built target ores.utility.lib); re-ran the deploy helper directly and confirmed the drop-in file already matched (no redundant reload). compass build --dry-run confirmed unaffected – no systemd-run wrapping shown, matching existing behaviour (no lock acquired, no slot known, nothing to wrap).

Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario, run through the QA Validation Runner panel) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

PRs

PR Title
#1862 [agile] Close systemd resource limits task, cascade story DONE
#1856 [compass] WS-7 remainder: memory-capped build slice
#1853 [compass] WS-4: per-environment tree/top views for compass services
#1849 [compass] WS-5 (unprivileged half): protect Emacs via systemd drop-in
#1845 [compass] WS-1: per-environment Claude slices with limits
#1842 [compass] WS-7 change 1: reduce build lock to two slots

Review

# Comment summary File Decision Notes
1 Docstring names the future drop-in dir "app-claude.slice.d/" (missing trailing dash), inconsistent with the actual dash-truncated template it describes compass_claude.py Fixed Corrected to "app-claude-.slice.d/".
2 Env names containing a literal dash would create an extra implicit intermediate slice level (harmless – the drop-in still applies at every level, and no current env name uses dashes) compass_claude.py Declined Genuinely latent, not hit by any real environment today (all use underscores); not worth a speculative guard for an input that can't currently occur.
3 _claude_slice_path is missing the intermediate app-claude.slice hierarchy segment – top would always point at a non-existent cgroup (flagged by all 3 review passes) compass_services.py Fixed Inserted the missing segment; re-verified live – top now reports a real row (48K, non-empty) instead of the previous empty output.
4 cmd_tree adds the Claude slice to –user-unit unconditionally, with no existence check unlike the fleet units – would abort the whole cgls call (including fleet subtrees) for any environment with no active Claude session compass_services.py Fixed Added _slice_is_loaded() (systemctl is-active without the .service-suffix mangling _unit_active_state applies) and gated the slice's inclusion on it; verified directly against both a real loaded slice and a made-up nonexistent one.
5 _has_user_systemd() in compass.py is a byte-for-byte duplicate of compass_claude._has_user_systemd() (flagged by all 3 review passes) compass.py Fixed Removed the duplicate; cmd_build now calls compass_claude._has_user_systemd() directly (compass_claude was already imported on the same code path for _env_name). Re-verified live (dry-run unaffected; real build still wraps correctly in the build slice, exit 0).
6 app-build-.slice.d/50-limits.conf omits TasksMax/ManagedOOMMemoryPressure=kill present in the sibling app-claude-.slice.d/ drop-in, with no comment explaining the difference 50-limits.conf Fixed Added a one-line rationale: a build's task count is bounded by -j (nowhere near TasksMax's headroom, unlike an open-ended interactive session), and pressure-based killing mid-build would produce a confusing partial/corrupted build rather than a clean MemoryMax-triggered failure.
7 Status table's "State | DONE" row trailing pipe not re-padded to the shorter value (cosmetic) task_systemd-per-environment-resource-limits.org Fixed Re-padded to match the other rows' column width.
8 #+updated header (2026-07-31) not bumped alongside "Last touched" (2026-08-05) task_systemd-per-environment-resource-limits.org Fixed Bumped #+updated to 2026-08-05.
9 "Waiting on" field doesn't say the WS-5 root-half wait is non-blocking, unlike the parent story's equivalent field task_systemd-per-environment-resource-limits.org Fixed Reworded to match story.org's phrasing: "left as a follow-up, not blocking this task's close."

Result

All unprivileged workstreams of the resource-management plan shipped across five PRs (#1842, #1845, #1849, #1853, #1856): WS-7 change 1 (two build-lock slots), WS-1 (per-environment Claude slices with MemoryHigh/MemoryMax/MemorySwapMax/CPUWeight), WS-5 unprivileged half (Emacs protection drop-in), WS-4 (compass services tree=/=top), and WS-7 remainder (memory-capped app-build-<env>.slice wrapping cmake --build). WS-2 and WS-3 were discovered already implemented predating this task (Sprint 24, PR #1806) – confirmed rather than reimplemented. All acceptance criteria met except the two explicitly out of this task's reach: WS-5's root half (needs root, not yet run) and the deliberate-OOM acceptance test (depends on the root half). WS-6 (PID 1 ownership) remains deferred, as scoped from the start.

Emacs 29.3 (Org mode 9.6.15)