How do I test the shell?

Table of Contents

The command unit's shape and file locations are in Shell entity patterns, and the recipe that creates a unit is How do I create shell commands for a new entity?.

1. Question

How do I test the shell?

2. Answer

The shell has three levels, and each answers a different question. Run the level that matches what you changed.

2.1. Level 1: the suites, with no fleet

./compass.sh test run --preset <preset> -- -R ores.shell

This runs six targets: ores.shell.api.tests, ores.shell.application.tests, ores.shell.analytics.tests, ores.shell.iam.tests, ores.shell.refdata.tests and ores.shell.trading.tests. They run in process and open no NATS connection, so they pass with the fleet down.

They answer "is the command wired?". A suite proves that a generated unit registers every derived verb, that each verb refuses to run without a session (You must be logged in) and that the output names the field it rejected. It cannot prove that a service answers the subject, because it never sends one. See How do I run the tests? for the per-component targets and the logging knobs.

2.2. Level 2: the script library, against a live fleet

Every recipe under doc/recipes/shell/<entity>/ tangles to one script per command under =projects/ores.shell/scripts/library/<category>/. Run one:

./compass.sh shell -l projects/ores.shell/scripts/library/<category>/<script>.ores

compass shell resolves the NATS URL, the TLS material, the subject prefix and the login from .env, then connects, logs in and runs the script. The options worth knowing:

  • --dry-run prints the exact ores.shell command and runs nothing. The password is masked.
  • -u and -p override the login. The defaults are ORES_SHELL_LOGIN_USERNAME and ORES_SHELL_LOGIN_PASSWORD.
  • --log-enabled and --log-level forward to the shell's own log file. The run also prints a progress log path.

This is the only level that answers "does a service answer this subject, and do the request and the response decode?". It needs the three prerequisites in How do I ready up an environment?: a current build, a current database and a provisioned tenant. Run How do I provision the system with Acme Corporation (holding group)? first to have a tenant to log in as.

2.3. Read the verdict, not the exit code

The exit code does not report the outcome. compass shell -l exits 0 even when a script aborts, so a wrapper that trusts it reports success on a broken script. Read the output:

  • {"result":{"outcome":"ok",...}} — the command reached a subscriber and the request decoded. This is the pass.
  • {"result":{"outcome":"missing",...}} — the key addresses no row. The generated recipes send the sentinel __none__, so this is what a fresh tenant answers.
  • {"result":{"outcome":"failed",...}} — the service refused the values. Read message before you call it a defect: a database validation function naming the sentinel is the command working as designed.
  • ✗ Script aborted at line N — the client rejected the input before the request left. Treat this as a failure and read the field it names.

The shell stops at the first failing command, so sweep the library one script per invocation. That is why the library keeps one script per command: a failure cannot hide the commands after it. With 40 scripts, expect a handful of missing and failed verdicts from the sentinels and at least one abort; the run passes when every verdict is explained, not when every verdict is ok.

2.4. Regenerate the scripts, never edit them

The .ores files are generated from the recipes:

./compass.sh build --direct tangle_shell_scripts

The Shell script drift CI job re-tangles and fails on a tracked script that no longer matches its recipe, so change the recipe and regenerate.

2.5. Level 3: the interactive REPL

./compass.sh shell

Type help for the command list. Type an entity name on its own to enter that entity's sub-REPL, then help again to list its verbs with their arguments — for example pricing_engine_types then help prints add <code> <description> <instrument_type_code> <reason> <commentary>. Use this level to learn a command's argument list and to reproduce a failure by hand before you change a recipe.

2.6. The inventory gate

doc/recipes/shell/shell.org is generated from the recipes, so a recipe that is not listed there is invisible to a reader browsing the inventory. Check it before you raise a PR:

python3 projects/ores.codegen/scripts/regenerate_shell_recipe_inventory.py --check

3. Script

No wrapper. Each level delegates to compass test run, compass shell or the inventory script, which is the point: the levels stay independent, so a failing level names its own cause.

4. Tested by

Manual: run all three levels on a feature branch that adds a shell command, and confirm that the level-2 run reaches a subscribing service and that the level-3 sub-REPL lists the new verb.

5. See also

Emacs 29.3 (Org mode 9.6.15)