Task: Notify super administrators of service restarts
Table of Contents
This page documents a task in the Record service lifecycle events and surface them in system health story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
A super administrator learns that a service restarted without watching the
services screen. When telemetry records a started event for a service that
has run before, it raises a service_restarted notification through the
inbox to the system tenant's super administrators. The notification names the
service, the instance and the host, and says whether the previous instance
stopped gracefully or was lost.
2. Status
| Field | Value |
|---|---|
| State | BACKLOG |
| Parent story | Record service lifecycle events and surface them in system health |
| Now | Not yet started. |
| Waiting on | Nothing. |
| Next | Begin implementation. |
| Last touched | 2026-10-06 |
3. Acceptance
- A
service_restartednotification kind is seeded, with its message in every locale the inbox carries. - Telemetry raises it to the holders of
telemetry::samples:readin the system tenant when a service starts after an earlier instance; the first start after a database is created raises nothing. - The notification says whether the previous instance stopped gracefully or was lost, and links to the services screen.
- Raising never fails the event's storage: an unsuccessful raise is logged.
- A test proves a restart raises one notification and a first start raises none.
4. Plan
(Implementation strategy. Written when work starts; key decisions
are distilled into the parent story's * Decisions at close, but the
plan itself stays — it is the historical record of what we did.)
5. Measurements
Every number this task claims about a tree, with the command that produced it
and the commit it was measured at. A number without both is not a measurement,
because a clean-up record describes a tree and trunk moves under a branch: a
consumer count, a gate's component count and a suite's size all change with no
code changing, and the reader of a bare number cannot tell which tree it came
from. Where the raw output matters, put it in an evidence/ file beside the
task and name that file in the Command column.
| What | Value | Command | Commit |
|---|---|---|---|
6. Notes
Settled with the operator on 2026-10-06: super administrators only, for now.
A raise reaches the permission holders of the raiser's own tenant
(ores_iam_current_tenant_id_fn() in notification_center.cpp), and
telemetry acts in the system tenant, so its raise reaches the super
administrators. Telling each tenant's administrators would need the raise to
fan out across tenants, which the inbox does not do; that stays out of scope.
A service cannot announce its own restart, because the inbox and IAM may not be up while it starts, so telemetry raises on the event it records.
7. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
8. PRs
| PR | Title |
|---|---|
9. Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|