Story: Log the fleet to the journal

Table of Contents

This page documents a story in Sprint 26. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

1. Goal

The fleet logs to the console and systemd captures it into the user journal, so one interface reaches every service and no file is written. compass stops reading a log file to decide whether a service is ready, because that file is the thing being removed, and it gains a verb that tails any unit including nats-server, with the last warnings and the last errors a filter away. The environment plugin shows that tail, and those warnings and errors, for one service at a time.

2. Status

Field Value
State DONE
Parent sprint Sprint 26
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-09-30

3. Acceptance

  • No service writes a log file, and every service's output reaches the user journal.
  • File logging is opt-in in ores.logging, matching the option the header already documents, and a unit that passes no filename writes no file.
  • compass decides a service is ready by reading the journal, and a running fleet still reports running rather than starting.
  • compass services logs prints a tail from the journal for any unit, including nats-server, which no selector reaches today.
  • The tail verb can return only the last warnings or only the last errors.
  • The Environment view opens a log overview for one service showing its tail and its warnings and errors.

4. Tasks

Task State Start End Description
Scaffold story: Log the fleet to the journal DONE 2026-09-30 2026-09-30 Story scaffolding rides this task: documents, sprint wiring, and the scaffold PR. Close it before merging that PR.
Log to the console and read readiness from the journal DONE   2026-09-30 Initial task for: Log the fleet to the journal
Add compass services logs over the journal DONE   2026-09-30 One verb that tails any service unit from the user journal, including nats-server, with filters for the last warnings and the last errors.
Add the per-service log overview to the environment plugin DONE   2026-09-30 A log overview for one service in the Environment view: the tail, and the last warnings and errors.

5. Decisions

6. Decisions

  • The console sink already exists, writes to std::cout, and sets auto_flush(true). Its own comment names podman logs, so console capture was already the intended path for a container. Nothing new is needed to log to the console; the flags were simply never passed.
  • File logging is meant to be opt-in and is not. logging_options.hpp already documents an empty filename as disabling file logging, but --log-filename carries a non-empty default, so the documented opt-out cannot be reached from the command line. Making the default empty is the fix, and it changes no caller.
  • The fleet logs three different ways today, which is why one interface is worth building. Twenty-three compiled units pass --log-enabled --log-level <level> --log-directory ../log --log-replica-index N and write their own file with the console sink off. ores.web is node and its unit redirects stdout and stderr to the same file with StandardOutput=append:. nats-server has no redirection and no logging keys in its config, so it already reaches the journal. A viewer written against files would serve two thirds of the fleet and miss nats.
  • Readiness reads the file that is being removed. _log_contains(ctx.log_dir / log_basename, "Service ready") is what separates running from starting, so dropping the file without moving that read would leave every active unit reading starting forever. The log read and the readiness read move in the same task, and that coupling fixes the order of the tasks.
  • nats-server is not in the service registry, so _resolve_service cannot reach it. A single log interface must accept a unit name, not only a registry selector.
  • journalctl --user is readable from an agent shell and returns this environment's unit output, so the journal is a usable source with no new permission.
  • The level stays per family even though the interface is single. ORES_SERVICE_LOG_LEVEL is baked into each compiled unit at generate time, ores.web reads ORES_WEB_LOG_LEVEL itself, and nats reads its own config.
  • Readiness needs the log far less than it first appears. Twenty-three of the twenty-four units are Type=notify, so the service calls sd_notify(READY=1) and systemd reporting the unit active is already the readiness signal; the Service ready scan adds nothing for them. Only ores.web is Type=simple and has no way to declare itself, and nats already has an ExecStartPost port check. So the readiness change is one unit rather than twenty-four, and it avoids a journal query per unit on every status call.
  • All three tasks ride one branch and one pull request, as the environment plugin did, because the plugin task cannot be verified without the verb and the verb has nothing to read until the fleet logs to the console.

7. Out of scope

  • Following a log live. The plugin's routes are one-shot JSON, so a streaming tail would change the transport rather than add a control.
  • Retention and rotation policy for the journal itself.
  • Interleaving every unit into one timeline.
  • Bringing logs back up from a remote or containerised deployment.

8. Result

The fleet logs to the console and systemd captures it into the user journal. No service writes a log file, and one verb reads any of them.

Three mechanisms became one. Twenty-three compiled units wrote their own file with the console sink off, ores.web's unit redirected stdout and stderr to that same file, and nats-server already reached the journal. Now the compiled units pass --log-to-console and no log directory, the node unit is left to systemd's own default, and nats needed nothing but the verb that can name it.

Compass reads the journal. Readiness stopped being a log question for twenty-three units, because they are Type=notify and systemd reports them active only after the service declared itself ready; only ores.web still reads a line, and it does log one. The start path polls ActiveState instead of a growing file, which deleted the byte-offset waiter and its snapshot.

File logging is opt-in, and the console is not. A caller that asks for logging and nothing else gets the console, which is what keeps compass shell --log-enabled working and stops the shell and the tools writing files nobody read.

compass services logs takes a service, a short name, a unit name or nats-server, and can return only the warnings or only the errors. The level is not journald's priority, which records all console output at one priority, so the filter reads the token the service printed: a compiled service formats timestamp [severity] [channel] message, and nats tags a line after its process id.

The environment plugin shows it. Every service row, nats included, opens a log overview that switches between the tail, the warnings and the errors, and says which kind of empty it is when there is nothing to show.

Proven by a fleet start on the new units: 22 of 24 reported running, nats reported running, nothing failed, and the journal held the services' own records in their own format. marketdata and synthetic read starting; they are almost certainly the schema drift rather than this change, because the database is a day behind HEAD. The fleet was stopped again afterwards.

Three defects came out of running it rather than reading it. The retry pass called itself, so an unstartable unit made a start loop forever. The console flag left an unused constant, which the build caught because warnings are errors here. And the first test run failed against a stale library, because the source was edited after the build had compiled it.

Emacs 29.3 (Org mode 9.6.15)