One scheduler draws every feed's gaps from its arrival law
Table of Contents
1. Summary
A synthetic feed has two separate jobs. One job decides what the next price
is. The other decides when it arrives. The
generator library holds the first
job: a value process whose next() returns a price. The scheduler holds the
second. Both live in ores.analytics.quant.
- The scheduler draws each series' gaps from its arrival law. The law is periodic, Poisson or Hawkes, and its gaps are in the market's operational time, so a busy market brings every series' ticks forward together. A periodic one gives equal gaps, so a feed can be fully regular. The other two give random and bursty gaps, as live ticks have.
- The value process is told how much time has passed. A bursty law then changes when prices are seen, and does not change the law of the price in real time.
2. Detail
2.1. The scheduler
One loop decides when every feed ticks: tick_scheduler in
service/tick_scheduler.hpp. It owns the market and a heap of series, and a
feed only registers its tick body with it at start() and leaves at stop().
The scheduler is a required dependency of every feed, so no feed keeps a clock
of its own.
The market scales every series by one bracket, the regime's multiplier plus
the heat shared over the enrolled rate. So every series shares one
operational time, tau, the integral of that bracket over wall time. The
operational_clock in domain/operational_clock.hpp keeps it: it owns the
regime chain, the heat and tau, advances tau in closed form between events,
and says when tau will reach a mark.
A series with sensitivity \(s\) whose law draws a gap \(g\) ticks when tau has moved by \(g / s\). That mark is fixed when the gap is drawn. A heat jump, a pin or an enrolment changes how fast tau moves and moves no mark, so the series wait in one heap keyed on their marks, a tick costs O(log n), and a change in the market reaches every series at the instant it happens.
The loop waits on a i_run_clock ( service/run_clock.hpp ): the steady clock
in a service, a manual clock in a test, which then asserts exact event times
without sleeping. A tick body is told the wall seconds since the series last
ticked and the wall time of the tick, both from that clock. A throw from a body
goes to a failure hook and the series keeps its place.
Only the loop changes the market or the heap. A pin, an added series or a
removed one from another thread is a command the loop applies at its clock's
now. remove and pin return once the loop has applied them, so a feed may
destroy what its body points at, and a reply to a pin shows the pin in force.
The system tenant can also change the market's numbers while it runs: the
chain's two rates, the heat's jump and decay, the market speed, and the tail
loading that tilts a spot price's shape with the pace (see the market volatility
channel note). configure
is a command like a pin. It applies at the loop's now and goes in the trace as a
configure event. What has passed stays as it was. The heat keeps its level and
fades at the new decay. The dwell in progress keeps its unit draw, so it ends at
the new rate's share of it. The calm multiplier is computed again, so the mean
pace stays at one. The clock refuses a set whose loop gain is one or more, a rate
or a speed that is not positive, or rates that leave no positive calm multiplier.
The refusal names the number, and then nothing changes.
The clock counts simulated seconds as it counts tau: each stretch of tau times
the speed in force while it passed. A tick's simulated seconds are the change in
that count, so a speed change reaches every spot feed at its next tick. A curve
fixes its step from the speed when it starts, because a scheme that only
approximates its law is checked against that step. It keeps that step until it
starts again, and it counts a tick's simulated seconds in steps of that length.
The settings are on the shell (synthetic regime set) and on the Market regime
tab, read only for a tenant.
The loop keeps two records a screen reads. Each heat reading, taken every 100 ms, carries the market's intensity λ(t): the ticks a second every series' law asks for at that instant, its intensity times its sensitivity times the pace. And an arrival history counts the ticks of each group by slot: an hour by the second and ten minutes by the tenth of a second, with the mean pace and the active share of each slot. A series names its group, and a feed names its asset class. The history is a count, not a list of ticks, so it costs the same at any tick rate; the Market screen folds it into a temporal raster, a column a minute and a cell a second of it, where a regime spell shows as a band of columns.
The scheduler holds no database, messaging or logging code, so it keeps the
rule of ores.analytics.quant. The synthetic side holds only the feeds. Any
other consumer can pace itself the same way without linking the synthetic
service.
The note Synthetic market data generation: approach lists the modes a user may choose and the control panel that sets them. This page says where the choice lives in the code.
2.2. The arrival laws
arrival_law in domain/arrival_law.hpp is a variant of three values. Each
carries all the state that decides its next gap, in the series' own
operational time.
periodic_lawgives the same gap every time, after a first gap that starts part way in, at a phase it draws. It is the default. A feed that wants no randomness uses it.poisson_lawgives an exponential gap (Poisson Process).hawkes_lawgives a gap that depends on the earlier ones (Hawkes Process). It starts at its stationary excess, so a run starts at its long-run rate.
draw_gap takes the law as const, and only record_event moves it, so a gap
that is drawn and not used leaves the law as it was. make_arrival_law builds
the law a spec names and throws on a mode it does not know. A silent fallback
to a periodic timer would hide a mistyped mode as a feed that quietly stays
regular.
Every draw comes from an rng_stream ( math/rng_stream.hpp ): Philox4x32-10
keyed by the run seed, the series key from the source name, and the purpose. A
series' draws do not depend on the thread, the order the feeds were made in,
or how many there are. The run seed is the service's --run-seed option, and
the service logs it when the run starts.
2.2.1. Why "periodic"
The obvious word, "steady", is taken. std::chrono::steady_clock and the asio
steady_timer mean a clock that never goes backwards, which says nothing about
how far apart the ticks are. "Fixed" is vague about what is fixed. A point
process with equal gaps is called periodic in the literature, and a reader of
the code meets the same word. The "Fixed interval" label in the approach note
is the periodic arrival law.
2.3. The parameters
A feed names its mode and one set of numbers.
| Mode | Parameters | Meaning |
|---|---|---|
| Periodic | ticks per hour | Equal gaps. |
| Poisson | ticks per hour | The mean rate. Gaps are exponential. |
| Hawkes | ticks per hour, branching ratio, decay | The long-run mean rate, how clustered the stream is, and how fast a burst fades. |
The Hawkes row uses the parametrisation in
Hawkes Process: the mean rate
\(\bar\lambda\), the branching ratio \(n\) and the decay \(\beta\). The factory turns
them into the baseline \(\mu = \bar\lambda(1-n)\) and the jump \(\alpha = n\beta\)
inside hawkes_law. A Hawkes law runs in operational time, so its branching
ratio is the same in a calm market and a busy one, and a burst fades faster in
wall time when the market is busy.
ticks_per_hour keeps its meaning in every mode. A person who switches from
Periodic to Hawkes keeps the same load on the pipeline, and changes only how
that load is spread in time. The approach note's \((\mu, \alpha, \beta)\) is the
same process in other coordinates.
The unit of an arrival law is the operational second, and ticks_per_hour is
the unit of the config. make_arrival_law converts once. At a pace of one, an
operational second is a wall second. The process has no notion of a unit, and a
template parameter would add noise for no gain.
2.4. How the elapsed time reaches the price
A gap can be half a tick or three ticks, so a value process cannot assume a step of one fixed length. The step takes the elapsed time:
i_stochastic_processhasnext(double elapsed).elapsedis the time since the last step, counted in nominal ticks, where one nominal tick is the tick length the process is configured for: the length of one period at the feed'sticks_per_hour. It has no unit and it needs no calendar.next()meansnext(1.0).- The feed turns the wall seconds since its last tick into nominal ticks by
dividing by the nominal period. A periodic series in a market at a pace of one
passes exactly one. A Poisson or Hawkes series at the same
ticks_per_hourpasses numbers whose long-run mean is one.
A process applies elapsed by the rule that is exact for its law:
| Process | Step for elapsed \(s\) |
|---|---|
| Geometric mixture | The component's log-return is \(N(m_k s,\ \sigma_k^2 s)\). Each component is a Brownian motion seen for \(s\) ticks. |
| Arithmetic mixture | The same, on the increment. |
| Ornstein-Uhlenbeck | The exact transition with \(\Delta t = s \cdot dt\). The formula holds for any step. |
The mixture rule is a modelling choice. Each regime is a Brownian motion, so means scale with \(s\) and variances scale with \(s\), and the weights stay as set. It is exact if the regime holds for the whole step. At \(s = 1\) each rule is one nominal tick, so a periodic feed with a given seed always steps the same sequence.
2.4.1. What the rule gives
The price process, in real time, does not depend on the tick times. A feed that ticks twice as often sees half-size steps, and the sum of those steps over an hour has the same law. Volatility per unit of real time is constant, so a burst of ticks shows many small moves and a lull shows few large ones. A Hawkes clock changes when the price is observed and leaves the price alone. This is also the limit of the model. In a live market bursts and large moves come together, and here they are independent. Linking the two is a different model.
2.4.2. Processes that do not take it
The short-rate processes of the yield curve path precompute their step from
dt. A lattice such as Black-Derman-Toy has steps fixed by its construction,
and the matrix processes (Heath-Jarrow-Morton, Libor market, the affine and
quadratic Gaussian models) precompute a covariance for one step. Only the three
engines above take next(elapsed). The others throw when called with an elapsed
other than one, and the validation refuses a non-periodic mode for a process
type that does not support it. The IR curve feed is periodic, and its
discount_factor(ticks_ahead) counts nominal ticks.
2.5. What validation guards
validate_process_parameters and its arrival companions
(validate_arrival_process_parameters, validate_step_compatibility) hold the
arrival rules. The service applies them when a feed is built; the editors offer
only what a process can follow, so the curve screens offer the periodic mode
alone.
- the rate is positive and finite;
- the decay is positive and finite;
- the branching ratio is at least zero and strictly below one;
- a non-periodic mode is allowed only for a process whose
supports_variable_step()answers true, whichvalidate_step_compatibilitydecides;
A branching ratio of one or more gives a stream that does not settle. A ratio very close to one gives gaps so short that the pipeline cannot keep up. The Hawkes process also stops after a fixed number of events in one request and reports that it did so, which guards the tick loop against a parameter set that the rules let through.
2.6. What the pipeline sees
Nothing changes after the feed. A tick carries its time, the marketdata ingest loop plans and stores it, and the rule in No code path may behave differently because a source is simulated holds: the arrival law is a property of the generator and no later stage reads it.
3. See also
- Market Arrival Regime — the market-wide factor: a calm/active state, a shared intensity and one operational clock per run, so the whole market can speed up together.
- Market Data Architecture — the structure note that orders this cluster, and where to read this page in it.
- The generator library — the value processes and the factory that builds them.
- The FX spot producer — a feed that runs on this scheduler.
- Synthetic market data generation: approach — the list of modes and the control panel.
- Stochastic Processes — the hub that carries the mathematics.
- Point Process and Conditional Intensity, Poisson Process and Hawkes Process — the mathematics of the three arrival processes.
- Ornstein-Uhlenbeck Process — its exact transition holds for any step.
- tick — the library whose Hawkes simulation inspired the implementation, and where we differ.