Story: Consolidate standalone scripts into compass
Table of Contents
This page documents a story in Sprint 19. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
Goal
Make compass the single, documented entry point for operating and
developing ORE Studio, folding the ~61 standalone shell and Python
scripts into compass subcommands organised by pillar. Today the
operational surface is a sprawl of scripts across build/scripts/,
projects/ores.sql/, projects/ores.codegen/, and elsewhere; each
reimplements .env sourcing, preset resolution, and path plumbing, and
they drift independently (the regroup broke one by changing the
directory layout it globbed). A contributor should run compass <pillar>
<verb> rather than memorising script paths.
This story supersedes and absorbs the service-registry consolidation task discovered during the regroup — that becomes one concrete migration within this larger effort.
Status
| Field | Value |
|---|---|
| State | STARTED |
| Parent sprint | Sprint 19 |
| Now | compass test results, sprint charts, branch staleness, bearings all shipped. rat now generates XML for compass test results. Windows env init bug scaffolded. compass build (Build pillar first slice) landed. |
| Waiting on | Nothing. |
| Next | Fix Windows venv activation path; document the compass env commands; then db/nats/Operate/Build/Generate migrations. |
| Last touched | 2026-06-05 |
Background: script inventory (2026-06-02)
61 operational/dev scripts, by category:
| Category | Count | Representative scripts |
|---|---|---|
| database | 19 | recreate_database.sh, setup_database.sh, run_sql.sh, reset_system.sh, service_vars.sh (generated), utility/* |
| codegen | 18 | codegen.sh, run_generator.sh, generate_*_schema.sh, generate_*_refdata.sh, validate_docs.sh, plantuml_er_generate.sh |
| services-lifecycle | 4 | start-services.sh, stop-services.sh, status-services.sh, start-client.sh |
| nats | 3 | nats.sh, init-nats.sh, generate_nats_certs.sh |
| environment-bootstrap | 3 | init-environment.sh, diff-environment.sh, install_debian_packages.sh |
| build | 3 | build.sh, collect_release_pr_data.py, generate_release_notes.py |
| pr-automation | 3 | pr-watch.sh, pr_watch.py, append_pr_to_backlog.py |
| site | 1 | serve-site.sh |
| ci-health | 1 | ci_health.py |
| other | 6 | clear-logs.sh, set_ssh_agent.sh (deleted 2026-06-06 — compass self-heals), setup_postgres_extensions.sh, compass itself, doc_*.py |
Most .sh scripts already share a .env-sourcing + ORES_PRESET idiom
(set -a; source .env; set +a), which is exactly the plumbing compass
already does once in Python.
Background: compass today
Compass is self-contained Python (projects/ores.compass/src/compass.py)
organised into five pillars — all of which serve the information
architecture, not system operation:
| Pillar | Purpose | Commands |
|---|---|---|
| Orient | where we are / fleet state | where, fleet, sprint, story |
| Search | retrieve docs (FTS/org-roam) | search=/=find, list, show |
| Scaffold | create branches + artefacts | goto, add |
| Capture | product-backlog triage | capture, inbox, next, deferred, backlog |
| Journal | per-worktree session log | journal |
Subcommands are registered in compass.py via a two-tier dispatch:
top-level commands with sub-subcommands are short-circuited before
argparse and handled by a cmd_<name>(argv) function with its own nested
ArgumentParser. Adding a pillar = add a cmd_<pillar>() handler + an
_EPILOG section + a short-circuit dispatch entry. Compass shells out
only for git=/=gh; everything else is native Python.
Proposed direction (to be finalised by the design task)
The existing five pillars cover navigating the project. The scripts cover running the project — a distinct concern that warrants new operational pillars. Candidate mapping (illustrative, not final):
| New pillar | Absorbs | Example commands |
|---|---|---|
| Provision | environment-bootstrap, nats setup, db create/teardown, certs, the service registry | compass env init, compass env diff, compass db recreate, compass nats init |
| Operate | services-lifecycle, nats CLI, logs | compass run services, compass run client, compass run status, compass logs clear |
| Build | build, site | compass build, compass site serve |
| Generate | codegen wrappers | compass gen schema, compass gen refdata (or keep delegating to ores.codegen) |
Open questions for the design task: pillar names and boundaries; whether
codegen folds in or stays a delegated tool (compass add already
delegates to it); whether scripts become thin shims that call compass
(backward-compat) or are deleted; how CI workflows that call the scripts
migrate; the single service registry's home (projects/services.json).
Acceptance
- The operational-pillar taxonomy — every operational script mapped to a
compass pillar + verb, with backward-compatibility and CI-migration
decisions — is settled in the design task and distilled into this
story's
* Decisions(kept inline, not indoc/plans/). - The operational pillars are implemented in
compassand documented in--helpand the compass recipes/knowledge docs. - Migrated scripts are removed (hard cutover) with every caller — CI workflows, the ores.lisp dashboard, recipes, docs — updated to call the compass verb. No shims, no dangling references.
- The service registry is consolidated to a single source (absorbs the
prior task);
init-environmentfunctionality runs under compass. cmake --build+ctestand a fullprovision → runround-trip pass.
Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| T1: Design the operational compass pillar taxonomy | DONE | 2026-06-02 | 2026-06-05 | System 2 design: 4 pillars + per-script mapping + Q1–Q8 decisions. Gates all migration tasks. |
| Consolidate service registries into a single source of truth | DONE | 2026-06-02 | 2026-06-02 | Confirmed + documented the codegen registry as the one source (D1: keep in codegen/models); recipe added. |
| Port init-environment to compass env init | DONE | 2026-06-02 | 2026-06-02 | Implement compass env init/diff/list (registry-backed, behaviour-preserving) + functional cutover (CI, dashboard, scripts; delete the scripts). |
| Document the compass env commands | BACKLOG | Update all recipes that referenced the env scripts + add a compass env recipe; enumerates every doc touched. | ||
| Add a command to record an env-version bump | BACKLOG | compass env bump: canonical version as data + changelog; init/validate read it. | ||
| Migrate db and nats setup into the Provision pillar | ABANDONED | 2026-06-05 | Absorbed by add_db_and_services_to_compass; nats half already shipped as compass nats. | |
| Migrate the Operate pillar (services, client, logs) | ABANDONED | 2026-06-05 | Absorbed by add_db_and_services_to_compass. | |
| Add db and services commands to compass | STARTED | 2026-06-05 | compass db {recreate,setup,drop,sql,reset-system,reset-tenant} + compass services {start,stop,status,clear-logs} + compass client; delete eleven scripts; repoint all callers. | |
| Migrate the Build pillar (build, site) | DONE | 2026-06-05 | 2026-06-05 | compass build (preset from .env; site/manual aliases) landed; compass site serve outstanding. |
| Auto-wire agile tables: add task, task start, task done | DONE | 2026-06-05 | 2026-06-05 | add task wires the BACKLOG row; task start/done flip row + dates; codegen takes –goal/–acceptance. |
| Add the Generate pillar (delegating gen verbs) | BACKLOG | compass gen verbs delegating to ores.codegen (keeps its own CLI). | ||
| Surface branch staleness vs main in compass Orient | STARTED | 2026-06-02 | Compact ↑ahead/↓behind-origin/main indicator (no fetch) on where/status/fleet, escalating to a warning when too stale. | |
| Port sprint_metrics.py to compass sprint charts | DONE | 2026-06-05 | compass sprint charts; auto-detects current sprint; deletes standalone script; updates recipes. | |
| Port parse_test_results.py to compass test results | DONE | 2026-06-05 | New test pillar; auto-resolves preset from .env; recipe for test run overview; deletes standalone script. | |
| Fix: compass test results missing after make rat | DONE | 2026-06-03 | 2026-06-03 | Wire XML reporter into rat custom targets so compass test results has data after the standard dev workflow. |
| Fix compass env init venv activation path on Windows | DONE | 2026-06-03 | 2026-06-05 | compass.sh sources venv/bin/activate on all platforms; Windows Git Bash needs venv/Scripts/activate. Detected from CI failure on PR #1031. |
| Add compass shell: run ores.shell with .env defaults | DONE | 2026-06-05 | 2026-06-05 | compass shell with .env defaults, -f scripted sessions; dashboard launcher rewired. PR 1097. |
| Add manual doc type to ores.codegen and compass | DONE | 2026-06-03 | 2026-06-03 | Template + type registration in codegen; default parent-dir and help text in compass; document_types.org and recipe updated. |
| Fold catalogue regeneration into compass | BACKLOG | Catalogues are derived data and should be regenerated, never hand-edited: the skills catalogue (build/scripts/generate_skills_catalogue.py, marker-delimited region, –check mode for CI), the runbooks catalogue, and the product-backlog indexes (regenerate_backlog_indexes.py) all follow the same scan-sources-and-rewrite-region pattern. Add a compass subcommand (e.g. compass build catalogues or a doc-sync verb) that regenerates all of them, and wire –check into CI so stale catalogues fail the build. | ||
| Delete set_ssh_agent.sh now compass self-heals SSH_AUTH_SOCK | DONE | 2026-06-06 | 2026-06-06 | compass.sh adopts the agent socket from ORES_SSH_AGENT_DIR when SSH_AUTH_SOCK is unset or dead (QoL task, DONE), so the standalone script is residue. Delete it and update every referer: the SSH memory, the pr-sync-branch skill, claude_code_settings.org (permission allowlist + prose) with a settings rebuild, and the script inventory rows. |
| Add compass db provision: wire ores.shell provisioning scripts into the DB lifecycle | BACKLOG | After compass db recreate the DB is empty (bootstrap-mode). Seed it via a new compass db provision subcommand that starts services if needed and invokes the appropriate ores.shell provisioning recipe (acme/barclays/smoke). Links: acme_system_provision, barclays_system_provision, bootstrap_smoke. |
Decisions
From the taxonomy design task (T1):
- Four operational pillars: Provision (env/db/nats setup), Operate (services lifecycle, client, logs), Build (compile, site), Generate (codegen delegation) — alongside the five info-architecture pillars.
- Noun-first grammar:
compass db recreate,compass services start, matching the existingcompass sprint status/compass story new. - Hard cutover: migrated scripts are deleted and all callers (CI, dashboard, recipes, docs) updated; no compatibility shims.
- Codegen stays its own CLI: the Generate pillar adds thin
compass genverbs that delegate toores.codegen; the 18 codegen scripts are not absorbed wholesale. - Logs belong to
services:compass services start --delete-logsfor the clean-run case (clears while down → no stale inodes), plus a standalonecompass services clear-logs. No separatelogsnoun. - Shared plumbing: one compass-side helper resolves
.env+ORES_PRESET+build/output/<preset>once, replacing the per-scriptset -a; source .env; set +aidiom. Single service registry source:projects/services.json. - Not migrated (out of scope): PR/CI tooling (
pr_watch,ci_health,append_pr_to_backlog), release tooling,set_ssh_agent.sh(since deleted — compass self-heals the agent socket; was: must be sourced), andores.codegen's own scripts.
The full script → pillar → verb mapping lives in the T1 design task.
Out of scope
- Rewriting the codegen engine itself (
ores.codegen); at most its wrapper scripts become compass verbs that delegate to it. - Changing what any script does — this is a relocation/consolidation onto a single CLI, not a behaviour change.
- The component regroup (done) and the org-mode model migration (separate story).
See also
- Compass CLI UX redesign — established the pillar model and the taxonomy-design-doc-first approach this story follows.
- Regroup C++ components under product-group parent directories — where the script-fragility (and the absorbed registry task) surfaced.