Task: Hear the events that move the rows

Table of Contents

This page documents a task in the Stream the changes as they happen story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

The change stream carries what the stores announce. One subscription hears every entity event, each message becomes the coarsest change that is true of it (this collection moved for this tenant), and the pump has something to drain on every tick.

2. Status

Field Value
State DONE
Parent story Stream the changes as they happen
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-31

3. Acceptance

  • One subscription hears every entity event, whatever its component or collection.
  • A received event records one change naming the tenant, the party when the event has one, and the collection.
  • An event that names no tenant records nothing, because no reader could be shown it.
  • The coalescer and the frame name the collection rather than a row key, because a key is the entity's own record and no two entities share one.
  • The feed runs beside the ticker in the server and stops when the server drains.
  • A unit test proves each mapping without a broker, and the live stream test proves an event arrives as a frame.

4. Plan

The first real producer of a change decides the shape of a change, so the shape changes here rather than being carried into the feed.

  • A subject names a collection and an action; it never names a row. The row key is the entity's own record, encoded by that entity's codec, so one subscription cannot decode the key of every entity. The change therefore names the component and the collection and nothing finer, and the coalescer drops the identifier it was scaffolded with. A client that hears a frame refetches the collection it draws; naming a row was an optimisation that no producer can supply.
  • Parsing runs from the end of the subject, because the delivered subject carries the deployment prefix: action, collection, version, component. The collection token drops its _events suffix, so it is the same name the route path uses.
  • The tenancy travels in the X-Tenant-Id and X-Party-Id headers, which the publisher sets. A message without a tenant is dropped: no reader could be shown it.
  • change_feed takes an ores::nats::message and records it, so a case builds a message and asserts the frame that a drain produces, with no broker in the test.
  • The server subscribes through the same registrar that already returns the subscriptions the service runner keeps alive, and runs the ticker from the runner's start hook. Both ends now exist, so the stream is no longer empty.
  • The pattern is *.v1.>. A NATS wildcard is a whole token, so the collection's _events suffix cannot be matched in the subscription and is filtered in the parse instead.

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
Core suite cases 45 ores.http.core.tests f32ce6b39f
Core suite assertions 187 ores.http.core.tests f32ce6b39f
Core suite cases before this task 36 ores.http.core.tests on 5be7a9469f 5be7a9469f
Live subscription line Hearing entity events on *.v1.> compass services logs ores.http.server f32ce6b39f
Live frames in one stream, 2 seconds 8, from 3 components curl -N -b <cookie> /api/v1/changes f32ce6b39f
Files main lost when #4293 merged 174 =git diff –diff-filter=D –name-only 9e500a223f^1 9e500a223f \ wc -l= 9e500a223f

6. Notes

The live proof found what the cases could not. The check is: sign a browser in, open /api/v1/changes, write something, read a frame. Three things blocked it in turn, and each was a real defect rather than a test gap.

First, nobody could sign in. Signing in answered Authentication is not configured although the log showed Fetched JWKS public key from IAM at startup: the door was built with server.verifier() before the runner installed the verifier. Fixed by Let the door reach the verifier.

$ curl -s -X POST http://127.0.0.1:21601/api/v1/auth/sessions \
    -H 'Content-Type: application/json' \
    -d '{"principal":"tenant_admin@acme_corporation","password":"…"}'
{"error":"Authentication is not configured"}

Second, the pattern heard nothing. With the door fixed, the stream opened and delivered no frame at all, and the pump logged nothing, so nothing arrived from the broker. The subscription was *.v1.*_events.>: a NATS wildcard is a whole token, so *_events is a literal, and the server accepts the subscription and delivers nothing to it. The pattern is now *.v1.> with the collection filtered in the parse, and a case walks the pattern token by token so the spelling cannot regress quietly. Captured as An events collection cannot be wildcarded.

Third, main had lost the work twice over. The build could not link because ores.iam.core called ores::synthetic::service::routed_code_count, which did not exist; main was fixed while I waited. Then the merged increments turned out to be gone from main entirely: merging #4293 deleted 174 files, including change_feed, the collection-shaped change and the door's verifier. Captured as A merge took an old tree over two merged increments, and restored by hotfix/restore-the-feed-and-the-door.

What the frames prove. A session opened as tenant_admin@acme_corporation, then a second sign-in. One stream carried three components' stores without the client asking anything: iam.login_info and iam.sessions from the sign-in, and marketdata.delivery_counts from the feed path. Every frame named its tenant and its component, and the party was empty for the tenant-wide ones.

What is still not proven: nothing in this task. The unit cases cover the mapping and the frame, and the deployment covers the composition.

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

8. PRs

PR Title
#4334 [repo] Put back the last doc records the reverted merge dropped
#4312 [repo] Restore everything the reverted merge dropped
#4303 [http] Restore the feed and the door, and fix the stream's pattern
#4287 [http] Hear the entity events the stores announce

9. Review

# Comment summary File Decision Notes
1 Local review of the restore-the-rest docs: no findings; misspell clean 4 task docs accept Each restored line matches a replay of every PR since 6b28050821 without the bad commit

10. Result

The stream now has a producer. change_feed turns one NATS message into one change: the subject gives the component, the collection and the action, read from the right so that the deployment's prefix does not matter, and the headers give the tenant and the party. A message with no tenant records nothing.

The first real producer also decided what a change is. A change names the collection and not the row, so change_event and coalesced_frame lost their identifier, the coalescer keeps a set of dirty collections instead of a map to identifiers, and a frame reads:

event: iam.tenant_types
data: {"tenant":"acme","party":"ops","component":"iam","entity":"tenant_types"}

The server subscribes once, in the same registrar that already returns the handles the runner keeps alive, and runs the ticker from the runner's start hook, so the hub is no longer empty. The window is 150 ms, the constant the design states.

The pattern the subscription uses is *.v1.>, not *.v1.*_events.>. A NATS wildcard is a whole token, so the second spelling is a literal that matches nothing; the server accepts it and delivers nothing, and the stream stayed silent with no error to read. The live run below is what found it, after the unit cases had passed the parser and the frames a hundred times. The feed rejects what is not an event in the parse instead, and the design states the cost of that.

Proven on a running deployment, once the session door could sign a browser in and the pattern could match: a signed-in caller opens /api/v1/changes, signs in a second time, and reads

event: iam.login_info
data: {"tenant":"55ae77e3-e255-4bde-8dc6-22ee2250fabb","party":"","component":"iam","entity":"login_info"}

event: iam.sessions
data: {"tenant":"55ae77e3-e255-4bde-8dc6-22ee2250fabb","party":"","component":"iam","entity":"sessions"}

event: marketdata.delivery_counts
data: {"tenant":"55ae77e3-e255-4bde-8dc6-22ee2250fabb","party":"3956198f-2b62-4bad-9047-5bfd2bfd0793","component":"marketdata","entity":"delivery_counts"}

followed by the keep-alive comment. Three components' stores reached one open stream without the client asking for anything.

Emacs 29.3 (Org mode 9.6.15)