Quality checks run between mapping and storage and ignore the source kind
Table of Contents
1. Summary
The service checks every mapped tick before it stores the tick. The checks are the same for a live source and a simulated source. A simulated source can produce a bad tick on purpose, so the checks are tested against real faults.
2. Detail
2.1. The checks
| Check | Drop reason | Rule |
|---|---|---|
| Mapping exists | unmapped |
The tick's id has a mapping for its provider. |
| Value is a number | not_a_number |
The text parses as a decimal. |
| Value is in range | out_of_range |
The value is inside the bounds set for the item. |
| Tick is fresh | stale |
The tick is not older than 5 seconds when the worker takes it. |
| Tick is newer | out_of_order |
The vendor time is not before the last stored vendor time of the series. |
A drop is counted for each adapter and each reason. The loop logs the first drop of a reason for an id and not each one.
2.2. Conflation
When several ticks of one key are in one batch, the loop keeps the last one. Conflation is a property of the batch and not of the source.
2.3. Why the kind does not matter
The simulator is the only source that can inject faults on demand. A check that skipped simulated ticks would hide the faults that the simulator exists to produce. See The simulator stops at the adapter seam.
3. See also
- Market Data Architecture — the structure note that orders this cluster.
- The ingest loop and the one pipeline — completes: the stage order in which these checks run.
- The simulator stops at the adapter seam — exemplifies: fault injection exercises these checks.