Story: Log the fleet to the journal
Table of Contents
This page documents a story in Sprint 26. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
1. Goal
The fleet logs to the console and systemd captures it into the user journal, so one interface reaches every service and no file is written. compass stops reading a log file to decide whether a service is ready, because that file is the thing being removed, and it gains a verb that tails any unit including nats-server, with the last warnings and the last errors a filter away. The environment plugin shows that tail, and those warnings and errors, for one service at a time.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent sprint | Sprint 26 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-09-30 |
3. Acceptance
- No service writes a log file, and every service's output reaches the user journal.
- File logging is opt-in in ores.logging, matching the option the header already documents, and a unit that passes no filename writes no file.
- compass decides a service is ready by reading the journal, and a running fleet still reports running rather than starting.
- compass services logs prints a tail from the journal for any unit, including nats-server, which no selector reaches today.
- The tail verb can return only the last warnings or only the last errors.
- The Environment view opens a log overview for one service showing its tail and its warnings and errors.
4. Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Scaffold story: Log the fleet to the journal | DONE | 2026-09-30 | 2026-09-30 | Story scaffolding rides this task: documents, sprint wiring, and the scaffold PR. Close it before merging that PR. |
| Log to the console and read readiness from the journal | DONE | 2026-09-30 | Initial task for: Log the fleet to the journal | |
| Add compass services logs over the journal | DONE | 2026-09-30 | One verb that tails any service unit from the user journal, including nats-server, with filters for the last warnings and the last errors. | |
| Add the per-service log overview to the environment plugin | DONE | 2026-09-30 | A log overview for one service in the Environment view: the tail, and the last warnings and errors. |
5. Decisions
6. Decisions
- The console sink already exists, writes to
std::cout, and setsauto_flush(true). Its own comment namespodman logs, so console capture was already the intended path for a container. Nothing new is needed to log to the console; the flags were simply never passed. - File logging is meant to be opt-in and is not.
logging_options.hppalready documents an empty filename as disabling file logging, but--log-filenamecarries a non-empty default, so the documented opt-out cannot be reached from the command line. Making the default empty is the fix, and it changes no caller. - The fleet logs three different ways today, which is why one interface is
worth building. Twenty-three compiled units pass
--log-enabled --log-level <level> --log-directory ../log --log-replica-index Nand write their own file with the console sink off.ores.webis node and its unit redirects stdout and stderr to the same file withStandardOutput=append:.nats-serverhas no redirection and no logging keys in its config, so it already reaches the journal. A viewer written against files would serve two thirds of the fleet and miss nats. - Readiness reads the file that is being removed.
_log_contains(ctx.log_dir / log_basename, "Service ready")is what separates running from starting, so dropping the file without moving that read would leave every active unit readingstartingforever. The log read and the readiness read move in the same task, and that coupling fixes the order of the tasks. nats-serveris not in the service registry, so_resolve_servicecannot reach it. A single log interface must accept a unit name, not only a registry selector.journalctl --useris readable from an agent shell and returns this environment's unit output, so the journal is a usable source with no new permission.- The level stays per family even though the interface is single.
ORES_SERVICE_LOG_LEVELis baked into each compiled unit at generate time,ores.webreadsORES_WEB_LOG_LEVELitself, and nats reads its own config. - Readiness needs the log far less than it first appears. Twenty-three of the
twenty-four units are
Type=notify, so the service callssd_notify(READY=1)and systemd reporting the unit active is already the readiness signal; theService readyscan adds nothing for them. Onlyores.webisType=simpleand has no way to declare itself, and nats already has anExecStartPostport check. So the readiness change is one unit rather than twenty-four, and it avoids a journal query per unit on every status call. - All three tasks ride one branch and one pull request, as the environment plugin did, because the plugin task cannot be verified without the verb and the verb has nothing to read until the fleet logs to the console.
7. Out of scope
- Following a log live. The plugin's routes are one-shot JSON, so a streaming tail would change the transport rather than add a control.
- Retention and rotation policy for the journal itself.
- Interleaving every unit into one timeline.
- Bringing logs back up from a remote or containerised deployment.
8. Result
The fleet logs to the console and systemd captures it into the user journal. No service writes a log file, and one verb reads any of them.
Three mechanisms became one. Twenty-three compiled units wrote their own file
with the console sink off, ores.web's unit redirected stdout and stderr to
that same file, and nats-server already reached the journal. Now the compiled
units pass --log-to-console and no log directory, the node unit is left to
systemd's own default, and nats needed nothing but the verb that can name it.
Compass reads the journal. Readiness stopped being a log question for
twenty-three units, because they are Type=notify and systemd reports them
active only after the service declared itself ready; only ores.web still
reads a line, and it does log one. The start path polls ActiveState instead of
a growing file, which deleted the byte-offset waiter and its snapshot.
File logging is opt-in, and the console is not. A caller that asks for logging
and nothing else gets the console, which is what keeps compass shell
--log-enabled working and stops the shell and the tools writing files nobody
read.
compass services logs takes a service, a short name, a unit name or
nats-server, and can return only the warnings or only the errors. The level is
not journald's priority, which records all console output at one priority, so
the filter reads the token the service printed: a compiled service formats
timestamp [severity] [channel] message, and nats tags a line after its
process id.
The environment plugin shows it. Every service row, nats included, opens a log overview that switches between the tail, the warnings and the errors, and says which kind of empty it is when there is nothing to show.
Proven by a fleet start on the new units: 22 of 24 reported running, nats
reported running, nothing failed, and the journal held the services' own
records in their own format. marketdata and synthetic read starting;
they are almost certainly the schema drift rather than this change, because
the database is a day behind HEAD. The fleet was stopped again afterwards.
Three defects came out of running it rather than reading it. The retry pass called itself, so an unstartable unit made a start loop forever. The console flag left an unused constant, which the build caught because warnings are errors here. And the first test run failed against a stale library, because the source was edited after the build had compiled it.