How do I test the shell?
Table of Contents
The command unit's shape and file locations are in Shell entity patterns, and the recipe that creates a unit is How do I create shell commands for a new entity?.
1. Question
How do I test the shell?
2. Answer
The shell has three levels, and each answers a different question. Run the level that matches what you changed.
2.1. Level 1: the suites, with no fleet
./compass.sh test run --preset <preset> -- -R ores.shell
This runs six targets: ores.shell.api.tests, ores.shell.application.tests,
ores.shell.analytics.tests, ores.shell.iam.tests,
ores.shell.refdata.tests and ores.shell.trading.tests. They run in process
and open no NATS connection, so they pass with the fleet down.
They answer "is the command wired?". A suite proves that a generated unit
registers every derived verb, that each verb refuses to run without a session
(You must be logged in) and that the output names the field it rejected. It
cannot prove that a service answers the subject, because it never sends one.
See How do I run the tests? for the per-component targets and
the logging knobs.
2.2. Level 2: the script library, against a live fleet
Every recipe under doc/recipes/shell/<entity>/ tangles to one script per
command under =projects/ores.shell/scripts/library/<category>/. Run one:
./compass.sh shell -l projects/ores.shell/scripts/library/<category>/<script>.ores
compass shell resolves the NATS URL, the TLS material, the subject prefix and
the login from .env, then connects, logs in and runs the script. The options
worth knowing:
--dry-runprints the exactores.shellcommand and runs nothing. The password is masked.-uand-poverride the login. The defaults areORES_SHELL_LOGIN_USERNAMEandORES_SHELL_LOGIN_PASSWORD.--log-enabledand--log-levelforward to the shell's own log file. The run also prints a progress log path.
This is the only level that answers "does a service answer this subject, and do the request and the response decode?". It needs the three prerequisites in How do I ready up an environment?: a current build, a current database and a provisioned tenant. Run How do I provision the system with Acme Corporation (holding group)? first to have a tenant to log in as.
2.3. Read the verdict, not the exit code
The exit code does not report the outcome. compass shell -l exits 0 even when
a script aborts, so a wrapper that trusts it reports success on a broken script.
Read the output:
{"result":{"outcome":"ok",...}}— the command reached a subscriber and the request decoded. This is the pass.{"result":{"outcome":"missing",...}}— the key addresses no row. The generated recipes send the sentinel__none__, so this is what a fresh tenant answers.{"result":{"outcome":"failed",...}}— the service refused the values. Readmessagebefore you call it a defect: a database validation function naming the sentinel is the command working as designed.✗ Script aborted at line N— the client rejected the input before the request left. Treat this as a failure and read the field it names.
The shell stops at the first failing command, so sweep the library one script
per invocation. That is why the library keeps one script per command: a failure
cannot hide the commands after it. With 40 scripts, expect a handful of missing
and failed verdicts from the sentinels and at least one abort; the run passes
when every verdict is explained, not when every verdict is ok.
2.4. Regenerate the scripts, never edit them
The .ores files are generated from the recipes:
./compass.sh build --direct tangle_shell_scripts
The Shell script drift CI job re-tangles and fails on a tracked script that no
longer matches its recipe, so change the recipe and regenerate.
2.5. Level 3: the interactive REPL
./compass.sh shell
Type help for the command list. Type an entity name on its own to enter that
entity's sub-REPL, then help again to list its verbs with their arguments —
for example pricing_engine_types then help prints
add <code> <description> <instrument_type_code> <reason> <commentary>. Use
this level to learn a command's argument list and to reproduce a failure by hand
before you change a recipe.
2.6. The inventory gate
doc/recipes/shell/shell.org is generated from the recipes, so a recipe that is
not listed there is invisible to a reader browsing the inventory. Check it
before you raise a PR:
python3 projects/ores.codegen/scripts/regenerate_shell_recipe_inventory.py --check
3. Script
No wrapper. Each level delegates to compass test run, compass shell or the
inventory script, which is the point: the levels stay independent, so a failing
level names its own cause.
4. Tested by
Manual: run all three levels on a feature branch that adds a shell command, and confirm that the level-2 run reaches a subscribing service and that the level-3 sub-REPL lists the new verb.
5. See also
- How do I run the tests?
- How do I ready up an environment?
- How do I provision the system with Acme Corporation (holding group)?
- How do I create shell commands for a new entity?
- How do I test a feature after implementation?
projects/ores.shell/scripts/README.org— the layout of the script library.