The ingest loop and the one pipeline

Source: market_data_tick_sequence.puml. Regenerate with plantuml -tpng doc/knowledge/architecture/market_data_tick_sequence.puml.

The stages, in order:

1. Receive. One wildcard subscription covers all adapter ingress subjects. A new adapter needs no subscription change. See The adapter subjects.
2. Map. The loop reads the tick's provider, id scheme, external id and field and finds the internal item. See The mapping belongs to the provider and not to the source. A tick with no mapping is dropped with the reason unmapped, and the loop logs it once for each id.
3. Check quality. See Quality checks run between mapping and storage and ignore the source kind.
4. Resolve the parties. The loop finds every party routed to the tick's source for that provider, for delivery. See A party routes to exactly one source for each provider.
5. Deliver and meter. The loop delivers the tick on each party stream and counts the delivery for each party. See Usage is recorded as events and priced later.
6. Compose. A composite item that uses this series is recomputed. See A composite attributes its use to the providers of its inputs.
7. Store. Asynchronously, in its next batch, the tick's lane writes one point in the source's stream, however many parties the source serves. A party's series points at the stream. The loop writes no observation row for a tick. A client reads the ticks with the history call, which checks the party's entitlement. See A source tick is stored once and entitled when it is read, A quote series is keyed by item, provider and party and The history call returns every tick of a range in one reply, or refuses the range.

• The subscription decodes a tick and puts it in a bounded queue. A full queue drops its oldest tick, because an old tick that waits while a newer one arrives is worse than a lost tick.
• Lanes sharded by stream take the ticks, so the ticks of one stream keep their order. A lane commits a batch when it holds a batch of rows or when its oldest row has waited the linger. See Ticks reach the store through sharded binary COPY in bounded batches.
• A batch holds the points of any stream, because a point carries no tenant. The point key is unique, so a redelivered tick is skipped. A batch that fails is split in two until the failing row is alone, and the loop logs that row and goes on.
• The worker drops a tick older than 5 seconds and counts the drop.
• The loop delivers a tick before its batch commits, with no wait for an acknowledgement or for the store. A slow store delays the history, never the live tick. A store that is down does not stop delivery: the lane keeps the ticks it cannot store, and it waits one second, then doubles the wait to thirty seconds, before it tries the store again.
• A shutdown stops the subscription, works through the queue for two seconds, and logs how many ticks it dropped.
• The loop keeps the id of each series after one read. The cache clears when routing changes and when a row for a series fails to store.

The loop caches the mapping and the routes. The service refreshes the cache when a mapping or a route changes, from the change events of those entities. The tick table has no change trigger and sends no notification, because a tick arrives many times each second and the party stream carries each value.

The ingress subject names the adapter, and the adapter id is the name of a source in the catalogue. The loop finds the source by that name, never by reading the text of the name. The source says the provider, so the tick is mapped by the vendor instruments and mappings of that provider. A route that names the source makes its party a consumer of the source. The tick is stored once, in the stream of the source, and delivered to each such party; the party's series under the provider points at that stream. A live source and a simulated source of one provider differ in their stream only.

The loop counts a drop for each adapter and each reason: unmapped, not_a_number, out_of_range, stale and out_of_order. The first drop of a reason for an adapter is logged. The bounds of out_of_range are one absolute limit for every item until items carry bounds of their own.

The CRM bridge, and any other consumer inside the service, is a tap registered for an instrument type. It reads the conflated live output of the lanes, so it sees the newest value and never a backlog. The loop has no if for an asset class. See The tick ingest pipeline.