Story: Consolidate standalone scripts into compass

Table of Contents

This page documents a story in Sprint 19. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

Goal

Make compass the single, documented entry point for operating and developing ORE Studio, folding the ~61 standalone shell and Python scripts into compass subcommands organised by pillar. Today the operational surface is a sprawl of scripts across build/scripts/, projects/ores.sql/, projects/ores.codegen/, and elsewhere; each reimplements .env sourcing, preset resolution, and path plumbing, and they drift independently (the regroup broke one by changing the directory layout it globbed). A contributor should run compass <pillar> <verb> rather than memorising script paths.

This story supersedes and absorbs the service-registry consolidation task discovered during the regroup — that becomes one concrete migration within this larger effort.

Status

Field Value
State STARTED
Parent sprint Sprint 19
Now compass test results, sprint charts, branch staleness, bearings all shipped. rat now generates XML for compass test results. Windows env init bug scaffolded. compass build (Build pillar first slice) landed.
Waiting on Nothing.
Next Fix Windows venv activation path; document the compass env commands; then db/nats/Operate/Build/Generate migrations.
Last touched 2026-06-05

Background: script inventory (2026-06-02)

61 operational/dev scripts, by category:

Category Count Representative scripts
database 19 recreate_database.sh, setup_database.sh, run_sql.sh, reset_system.sh, service_vars.sh (generated), utility/*
codegen 18 codegen.sh, run_generator.sh, generate_*_schema.sh, generate_*_refdata.sh, validate_docs.sh, plantuml_er_generate.sh
services-lifecycle 4 start-services.sh, stop-services.sh, status-services.sh, start-client.sh
nats 3 nats.sh, init-nats.sh, generate_nats_certs.sh
environment-bootstrap 3 init-environment.sh, diff-environment.sh, install_debian_packages.sh
build 3 build.sh, collect_release_pr_data.py, generate_release_notes.py
pr-automation 3 pr-watch.sh, pr_watch.py, append_pr_to_backlog.py
site 1 serve-site.sh
ci-health 1 ci_health.py
other 6 clear-logs.sh, set_ssh_agent.sh (deleted 2026-06-06 — compass self-heals), setup_postgres_extensions.sh, compass itself, doc_*.py

Most .sh scripts already share a .env-sourcing + ORES_PRESET idiom (set -a; source .env; set +a), which is exactly the plumbing compass already does once in Python.

Background: compass today

Compass is self-contained Python (projects/ores.compass/src/compass.py) organised into five pillars — all of which serve the information architecture, not system operation:

Pillar Purpose Commands
Orient where we are / fleet state where, fleet, sprint, story
Search retrieve docs (FTS/org-roam) search=/=find, list, show
Scaffold create branches + artefacts goto, add
Capture product-backlog triage capture, inbox, next, deferred, backlog
Journal per-worktree session log journal

Subcommands are registered in compass.py via a two-tier dispatch: top-level commands with sub-subcommands are short-circuited before argparse and handled by a cmd_<name>(argv) function with its own nested ArgumentParser. Adding a pillar = add a cmd_<pillar>() handler + an _EPILOG section + a short-circuit dispatch entry. Compass shells out only for git=/=gh; everything else is native Python.

Proposed direction (to be finalised by the design task)

The existing five pillars cover navigating the project. The scripts cover running the project — a distinct concern that warrants new operational pillars. Candidate mapping (illustrative, not final):

New pillar Absorbs Example commands
Provision environment-bootstrap, nats setup, db create/teardown, certs, the service registry compass env init, compass env diff, compass db recreate, compass nats init
Operate services-lifecycle, nats CLI, logs compass run services, compass run client, compass run status, compass logs clear
Build build, site compass build, compass site serve
Generate codegen wrappers compass gen schema, compass gen refdata (or keep delegating to ores.codegen)

Open questions for the design task: pillar names and boundaries; whether codegen folds in or stays a delegated tool (compass add already delegates to it); whether scripts become thin shims that call compass (backward-compat) or are deleted; how CI workflows that call the scripts migrate; the single service registry's home (projects/services.json).

Acceptance

  • The operational-pillar taxonomy — every operational script mapped to a compass pillar + verb, with backward-compatibility and CI-migration decisions — is settled in the design task and distilled into this story's * Decisions (kept inline, not in doc/plans/).
  • The operational pillars are implemented in compass and documented in --help and the compass recipes/knowledge docs.
  • Migrated scripts are removed (hard cutover) with every caller — CI workflows, the ores.lisp dashboard, recipes, docs — updated to call the compass verb. No shims, no dangling references.
  • The service registry is consolidated to a single source (absorbs the prior task); init-environment functionality runs under compass.
  • cmake --build + ctest and a full provision → run round-trip pass.

Tasks

Task State Start End Description
T1: Design the operational compass pillar taxonomy DONE 2026-06-02 2026-06-05 System 2 design: 4 pillars + per-script mapping + Q1–Q8 decisions. Gates all migration tasks.
Consolidate service registries into a single source of truth DONE 2026-06-02 2026-06-02 Confirmed + documented the codegen registry as the one source (D1: keep in codegen/models); recipe added.
Port init-environment to compass env init DONE 2026-06-02 2026-06-02 Implement compass env init/diff/list (registry-backed, behaviour-preserving) + functional cutover (CI, dashboard, scripts; delete the scripts).
Document the compass env commands BACKLOG     Update all recipes that referenced the env scripts + add a compass env recipe; enumerates every doc touched.
Add a command to record an env-version bump BACKLOG     compass env bump: canonical version as data + changelog; init/validate read it.
Migrate db and nats setup into the Provision pillar ABANDONED   2026-06-05 Absorbed by add_db_and_services_to_compass; nats half already shipped as compass nats.
Migrate the Operate pillar (services, client, logs) ABANDONED   2026-06-05 Absorbed by add_db_and_services_to_compass.
Add db and services commands to compass STARTED 2026-06-05   compass db {recreate,setup,drop,sql,reset-system,reset-tenant} + compass services {start,stop,status,clear-logs} + compass client; delete eleven scripts; repoint all callers.
Migrate the Build pillar (build, site) DONE 2026-06-05 2026-06-05 compass build (preset from .env; site/manual aliases) landed; compass site serve outstanding.
Auto-wire agile tables: add task, task start, task done DONE 2026-06-05 2026-06-05 add task wires the BACKLOG row; task start/done flip row + dates; codegen takes –goal/–acceptance.
Add the Generate pillar (delegating gen verbs) BACKLOG     compass gen verbs delegating to ores.codegen (keeps its own CLI).
Surface branch staleness vs main in compass Orient STARTED 2026-06-02   Compact ↑ahead/↓behind-origin/main indicator (no fetch) on where/status/fleet, escalating to a warning when too stale.
Port sprint_metrics.py to compass sprint charts DONE   2026-06-05 compass sprint charts; auto-detects current sprint; deletes standalone script; updates recipes.
Port parse_test_results.py to compass test results DONE   2026-06-05 New test pillar; auto-resolves preset from .env; recipe for test run overview; deletes standalone script.
Fix: compass test results missing after make rat DONE 2026-06-03 2026-06-03 Wire XML reporter into rat custom targets so compass test results has data after the standard dev workflow.
Fix compass env init venv activation path on Windows DONE 2026-06-03 2026-06-05 compass.sh sources venv/bin/activate on all platforms; Windows Git Bash needs venv/Scripts/activate. Detected from CI failure on PR #1031.
Add compass shell: run ores.shell with .env defaults DONE 2026-06-05 2026-06-05 compass shell with .env defaults, -f scripted sessions; dashboard launcher rewired. PR 1097.
Add manual doc type to ores.codegen and compass DONE 2026-06-03 2026-06-03 Template + type registration in codegen; default parent-dir and help text in compass; document_types.org and recipe updated.
Fold catalogue regeneration into compass BACKLOG     Catalogues are derived data and should be regenerated, never hand-edited: the skills catalogue (build/scripts/generate_skills_catalogue.py, marker-delimited region, –check mode for CI), the runbooks catalogue, and the product-backlog indexes (regenerate_backlog_indexes.py) all follow the same scan-sources-and-rewrite-region pattern. Add a compass subcommand (e.g. compass build catalogues or a doc-sync verb) that regenerates all of them, and wire –check into CI so stale catalogues fail the build.
Delete set_ssh_agent.sh now compass self-heals SSH_AUTH_SOCK DONE 2026-06-06 2026-06-06 compass.sh adopts the agent socket from ORES_SSH_AGENT_DIR when SSH_AUTH_SOCK is unset or dead (QoL task, DONE), so the standalone script is residue. Delete it and update every referer: the SSH memory, the pr-sync-branch skill, claude_code_settings.org (permission allowlist + prose) with a settings rebuild, and the script inventory rows.
Add compass db provision: wire ores.shell provisioning scripts into the DB lifecycle BACKLOG     After compass db recreate the DB is empty (bootstrap-mode). Seed it via a new compass db provision subcommand that starts services if needed and invokes the appropriate ores.shell provisioning recipe (acme/barclays/smoke). Links: acme_system_provision, barclays_system_provision, bootstrap_smoke.

Decisions

From the taxonomy design task (T1):

  • Four operational pillars: Provision (env/db/nats setup), Operate (services lifecycle, client, logs), Build (compile, site), Generate (codegen delegation) — alongside the five info-architecture pillars.
  • Noun-first grammar: compass db recreate, compass services start, matching the existing compass sprint status / compass story new.
  • Hard cutover: migrated scripts are deleted and all callers (CI, dashboard, recipes, docs) updated; no compatibility shims.
  • Codegen stays its own CLI: the Generate pillar adds thin compass gen verbs that delegate to ores.codegen; the 18 codegen scripts are not absorbed wholesale.
  • Logs belong to services: compass services start --delete-logs for the clean-run case (clears while down → no stale inodes), plus a standalone compass services clear-logs. No separate logs noun.
  • Shared plumbing: one compass-side helper resolves .env + ORES_PRESET + build/output/<preset> once, replacing the per-script set -a; source .env; set +a idiom. Single service registry source: projects/services.json.
  • Not migrated (out of scope): PR/CI tooling (pr_watch, ci_health, append_pr_to_backlog), release tooling, set_ssh_agent.sh (since deleted — compass self-heals the agent socket; was: must be sourced), and ores.codegen's own scripts.

The full script → pillar → verb mapping lives in the T1 design task.

Out of scope

  • Rewriting the codegen engine itself (ores.codegen); at most its wrapper scripts become compass verbs that delegate to it.
  • Changing what any script does — this is a relocation/consolidation onto a single CLI, not a behaviour change.
  • The component regroup (done) and the org-mode model migration (separate story).

See also

Emacs 29.1 (Org mode 9.6.6)