The tick ingest pipeline: freshness first

The NATS callback does four things and no input or output. It decodes the tick. It looks up the policy of the class of the tick. It checks the age of the tick. It writes the tick into the ring of its lane. A lane is chosen by a hash of the series key. A tick of one series always reaches the same lane, and so do all quotes of one curve. The order of a series holds without a lock, and an operation on a whole curve stays inside one lane.

Parsing an oresmd URI costs more than the rest of the receive stage. The receiver keeps a table that maps a URI string to its parsed datum, its series key and its class. The codec runs once for each datum, and a later tick of the datum costs one lookup.

Each lane owns one ring buffer of fixed size. The slots are allocated once, and the ring is a power of two in size. A sequence number marks each slot, in the style of the Disruptor. The receiver claims the next slot and publishes it. The lane worker reads up to the newest published slot.

The ring never blocks the receiver. The receiver writes the next slot whether or not the worker has read the slot before it. The worker checks the sequence number of each slot that it reads. When the receiver has lapped the worker, the numbers show it. The worker skips to the oldest slot that is still valid, and it adds the number of skipped slots to a counter of dropped ticks. The newest tick always wins, and the writer never touches a slot that the worker is reading. A starting size is 65,536 slots for each lane. At 10,000 ticks each second in one lane, the ring holds about six seconds.

Every tick has an age: the time now minus the observation_time of the tick. The receiver drops a tick that is older than the limit of its class, and it counts the drop. The lane worker checks again when it takes the tick from the ring. A starting limit is 5 seconds for a live tick. The limit is a setting for each asset class.

The gate changes the order of service in one more way. A lane of a perishable class that is behind takes the newest ticks first and drops the old ones. It does not work through the old ticks.

A lane worker keeps the newest tick of each datum in the batch it took. It publishes those ticks to the live consumers on the party subjects. The publish uses core NATS and waits for no acknowledgement. The live path ends here, and it never waits for the store.

The class sets a conflation interval. The lane publishes a datum at most once in each interval, and the lane keeps the newest tick of the datum until the interval ends. The lane publishes that last tick at the end of the interval even when no newer tick arrives, so no value waits longer than the interval. A class with consistency of series publishes the changed quotes of one series together at the end of a cycle.

A consumer selects what it watches on the broker, so the live subject splits its tail into tokens:

marketdata.v1.ops.market_tick.<tenant>.<party>.<instrument_type>.<series_token>.<coordinate_token>

The series token is the percent-encoded series URI. The coordinate token is the percent-encoded coordinates, or one fixed character for a series without any. A consumer then subscribes to a class with ...<instrument_type>.>, to a series with ...<instrument_type>.<series_token>.>, or to one datum. A chart subscribes to the series that it shows, and the broker sends nothing else.

The ingress subject is marketdata.v1.ingress.<adapter>.<id token>.

Each lane keeps the recent ticks of its datums in memory. This is the real-time tier of a tickerplant system. A tick enters the window when the lane worker takes it from the ring, so the window is as fresh as the live path and does not wait for the store.

• What it holds. The last tick of every datum, always. For a datum that a consumer watches, it also holds a bounded run of recent ticks. A starting bound is the last 600 ticks or the last 10 minutes, whichever is smaller. A datum with no watcher keeps only its last tick, so the memory use follows the number of watched datums and not the number of all datums.
• Who reads it. The history call reads it. A chart that opens or reconnects, and a consumer that joins late, call for the ticks of a span that ends now. The lane that owns the datum answers the part of the span that the window covers from memory. The read checks the tenant and the party of the caller against the routes of the datum, as any read of the store does.
• How a read reaches it. The lane worker is the only writer of the window. A read is a request that the lane worker answers between two cycles from the window of its own lane. The reader never touches the window, so the single writer rule holds.
• Where it ends. A span older than the window comes from the store, and the history call joins the two in one reply. See The history call returns every tick of a range in one reply, or refuses the range.
• What it can differ on. The window holds every tick that the lane took, and the store may hold fewer when a lane is catching up or shedding. A reader that sees both prefers the window for the span that it covers.
• Restart. The window is empty after a restart and fills as ticks arrive. A chart reads the store for the span before that.

The same cycle writes every tick that passed the age gate in one transaction, with a multi-row insert or a COPY. One commit covers hundreds or thousands of rows. The cost of the commit is shared by all of them.

• The series identity of each key comes from a cache in memory. A route event clears the cache. A series that does not exist yet is created on a slow path, outside the batch.
• The shape check and the manual point check run once for the batch.
• A batch that fails is split in two and tried again. The split continues until it isolates the row that the database refuses. The lane drops that row, counts it, and logs it.
• The size of a batch has a cap. A starting value is 5,000 rows.

A lane measures its lag from the oldest tick that it processes. The lag selects one of three states.

Source: tick_ingest_overload_states.puml. Regenerate with plantuml -tpng doc/knowledge/architecture/tick_ingest_overload_states.puml.

• Live. The lag is small. The lane writes every tick.
• Catching up. The lag is above the first limit. The lane writes only the newest tick of each datum for each second, or one bar for each datum for each second when its class keeps the extremes. The shape of the history stays, and the volume falls.
• Shedding. The lag is above the second limit. The lane drops every tick older than the age limit and counts each drop.

A store that does not answer counts as an infinite lag. The lane keeps its live path and its window. It holds a bounded number of batches, and after the second limit it sheds. It tries the connection again with a growing gap, and it writes the newest ticks first when the store returns. A shutdown flushes the open batch within a set time.

The lane goes back one state when the lag falls below half of the limit. The gap stops the lane from flipping between two states. Starting values are 5 seconds for the first limit and 30 seconds for the second. Tuning by measurement sets the final values.

Source: tick_ingest_lane_cycle.puml. Regenerate with plantuml -tpng doc/knowledge/architecture/tick_ingest_lane_cycle.puml.

Each lane reports these values to the status log and to telemetry.

• The depth of the ring and the age of its oldest slot.
• The lag from the tick time to the commit, and from the tick time to the live publish.
• The size of each batch and the time of each commit.
• The ticks dropped, by source and by reason: the ring was lapped, the tick was too old, the datum was thinned, or the database refused the row.
• The share of the time that the worker is busy, for each lane, so that an uneven split of series over lanes shows.

An alert fires when the lag is above the first limit for longer than a set time, so a lag of seconds shows within seconds.

• A datum is one quote. Its URI names the instrument and every coordinate, such as the term of a swap.
• A series is the datum without its coordinates. The schema row of an instrument type says which fields are identity and which are coordinates. A curve whose pillars are coordinates is one series with many datums. A curve whose tenor is an identity field is many series.
• A lane key is the series. All quotes of a curve share a lane.
• Conflation, the recent window and thinning work on the datum. Ordering works on the series.

A tick class is an instrument type, such as fx_spot or ir_swap. A policy table holds one row for each class, and a default row covers every class without a row. A lane reads the table as an immutable snapshot, and a change reaches the lane as a message. The fields of a row are these.

A new asset class works with the default row. It needs a row only when a field differs. That keeps the property that a new kind needs no change in the ingest. An equity feed would set bars, because its high and low matter. An interest rate curve would set a longer age_limit, because it ticks slowly, and consistency of series, because a curve must not publish half old and half new quotes. A trade feed would set durable and conflation of none.

A quote is perishable, and the rules above apply to it. Some ticks are not perishable. A trade, an official fixing and a close must not be lost, and an old one is still correct. A class of durable ticks changes the rules in four ways.

• The age gate does not apply.
• Conflation is none.
• The lane never sheds. A full ring makes the receiver wait, and the producer feels the back-pressure.
• The lane stores the batch before it publishes the batch to live consumers.

A durable tick never shares the receiver of the quotes. A full durable ring would make its receiver wait, and every quote behind it on the same thread would wait too. Durable ticks use their own entry: a JetStream consumer on its own thread that acknowledges a tick after the commit. The consumer feeds lanes of its own. Back-pressure reaches the producer through the acknowledgements, and a restart replays what the consumer did not acknowledge. Both tiers share the lane code, the batch writer and the gauges.

The first durable class is the fixing, which public sources publish once a day. A fixing skips the age check, takes a queue of its own that never sheds, and is stored in the fixings table, which keeps it for good. It does not yet take the JetStream entry above. See Public sources publish daily fixings through an adapter.

Some consumers sit inside the service. The CRM bridge turns an FX spot rate into a driver update. Cross rates and blended rates are derived from observed rates. The pipeline serves them through taps.

• A tap is registered for an instrument type. It reads the conflated live output of the lanes, so it sees the newest value and never a backlog. The CRM bridge fits this. It keeps one rate and one observation time for each pair of each party, and it asks for no more.
• A tap runs outside the lanes, because a derived value can depend on inputs from several lanes.
• A derived value enters the pipeline again as a tick with its own source and source kind derived. It follows the same rules as any other tick.

A tick has three times. The producer stamps the observation time. The receiver stamps the arrival time. The store records the commit time.

• The age of a tick is now minus the observation time. The age limit uses it.
• The transport lag is the arrival time minus the observation time. A large lag, or a negative one, points at clock skew between the producer and the service.
• A tick stamped in the future by more than a set allowance takes the arrival time as its observation time, and the lane counts it.
• A tick older than the newest tick of its datum is not published live. The store keeps it, because it is history.

Two different conditions look alike to a consumer. A tick that is old is a lag. A source that sends no tick is silence. The age limit handles the first. The second needs its own rule. The pipeline keeps the time of the last tick of each source, and it reports a source as silent after the silence_after of its class. Hedgr shows a stale verdict, and the CRM bridge reports a rate as stale after 15 minutes and disconnected after 60 minutes. Both are silence rules. They stay with their consumers. The pipeline gives them the time of the last tick and the lag of the source, so they need not guess.

The age gate and the shedding apply to live ticks only. An import of ORE market data and a backfill from a vendor are bulk loads. They use the batch writer through their own entry. They carry old observation times on purpose, and no gate drops them. The entry controls its own pace, so a bulk load does not starve the live lanes. A vendor snapshot that follows a reconnect is a live tick, with its own time.

• A hot source cannot fill a ring for the others. The conflation interval limits the live rate of each datum. The lane counts drops for each source, so a hot source shows by name.
• A tick that fails to decode, or that names no datum, is counted and dropped at the receiver. It never reaches a lane.
• A lane catches an exception in a cycle, logs it, counts it, and goes on. A lane that stops is restarted, and its ring is kept.
• The tenants share lanes. A write that fails for one tenant is split off by the bisect step and does not fail the batch of the others.

The lane keeps one last tick for each datum. At a million datums and 100 bytes for each, that is about 100 MB, which is an estimate to check. Windows exist only for watched datums. The ring and the window are bounded, so the memory of a lane does not grow with the tick rate.

A producer follows these rules, whatever its asset class.

1. Publish a market_tick on the producer subject. Name the datum by its canonical oresmd URI.
2. Stamp the observation time from the clock of the source. Add a sequence number for each source, so that the receiver counts the ticks that it never saw.
3. Publish a quote without waiting for an acknowledgement.
4. Publish a series from one producer only, as the rule of series exclusivity says.
5. Use an existing instrument type, or add one. A new instrument type works with the default policy row.
6. Declare the expected rate and the number of datums, so that capacity is planned.
7. State the durability tier. A quote needs no statement.
8. Add a policy row only when a default field is wrong for the class.

A tick of any class reaches live consumers within the age limit of its class, or the pipeline counts it as dropped, by source and by reason. The newest tick of a datum is never behind an older one. A feed needs no code in the pipeline.

One process holds many lanes. A single process has limits: one NATS delivery thread decodes every tick, and one database serves every lane. When these bind, run several ingest instances. Each takes a share of the sources, which a setting assigns, and the series key keeps one series in one instance. The lane design does not change.

The core of a lane is a pure function of a batch, a policy and a clock. A test injects the clock and checks the rules without a broker: no tick older than its limit is published, the newest tick of a datum is never behind an older one, the memory stays bounded, and the states change at their limits. Failure injection covers a slow commit, a store that is down, a lane that throws, and a full ring. The tick injector is the load test. It becomes a tool in the repository, so that anyone can reproduce a lag.

• One append-only hypertable of points for each retention class. The columns are the stream, ts, coord and value. It has no tenant column, no trigger, no valid_to, a unique key on stream, time and coordinate, and no row-level security. Retention drops whole chunks by time, so a class with a different retention needs its own table. The number of tables follows the policy table and does not grow with the tenants.
• Compression. Closed chunks are compressed, segmented by stream and ordered by time. See The tick store is bounded.
• Grants, not policies. Only the marketdata service's roles hold grants on the streams and the points. A tenant's session holds none.
• A writer role with INSERT only. It writes with binary COPY and has no SELECT. See Ticks reach the store through sharded binary COPY in bounded batches.
• Readers check entitlement. The history read checks, in the party's tenant and under row-level security, that the party's series names the stream and that the party may read it, then reads the points with the service's own role.
• Hot reads skip the table. A chart and the latest value come from the recent window and the live subject. The subject has the tenant and the party.
• A derived series enters through the same door. The blending service reads its inputs on the live path, in the context of the party. It emits ticks for the party's derived stream into the same lane and the same writer.