One scheduler draws every feed's gaps from its arrival law

Table of Contents

1. Summary

A synthetic feed has two separate jobs. One job decides what the next price is. The other decides when it arrives. The generator library holds the first job: a value process whose next() returns a price. The scheduler holds the second. Both live in ores.analytics.quant.

  1. The scheduler draws each series' gaps from its arrival law. The law is periodic, Poisson or Hawkes, and its gaps are in the market's operational time, so a busy market brings every series' ticks forward together. A periodic one gives equal gaps, so a feed can be fully regular. The other two give random and bursty gaps, as live ticks have.
  2. The value process is told how much time has passed. A bursty law then changes when prices are seen, and does not change the law of the price in real time.

2. Detail

2.1. The scheduler

One loop decides when every feed ticks: tick_scheduler in service/tick_scheduler.hpp. It owns the market and a heap of series, and a feed only registers its tick body with it at start() and leaves at stop(). The scheduler is a required dependency of every feed, so no feed keeps a clock of its own.

The market scales every series by one bracket, the regime's multiplier plus the heat shared over the enrolled rate. So every series shares one operational time, tau, the integral of that bracket over wall time. The operational_clock in domain/operational_clock.hpp keeps it: it owns the regime chain, the heat and tau, advances tau in closed form between events, and says when tau will reach a mark.

A series with sensitivity \(s\) whose law draws a gap \(g\) ticks when tau has moved by \(g / s\). That mark is fixed when the gap is drawn. A heat jump, a pin or an enrolment changes how fast tau moves and moves no mark, so the series wait in one heap keyed on their marks, a tick costs O(log n), and a change in the market reaches every series at the instant it happens.

The loop waits on a i_run_clock ( service/run_clock.hpp ): the steady clock in a service, a manual clock in a test, which then asserts exact event times without sleeping. A tick body is told the wall seconds since the series last ticked and the wall time of the tick, both from that clock. A throw from a body goes to a failure hook and the series keeps its place.

Only the loop changes the market or the heap. A pin, an added series or a removed one from another thread is a command the loop applies at its clock's now. remove and pin return once the loop has applied them, so a feed may destroy what its body points at, and a reply to a pin shows the pin in force.

The system tenant can also change the market's numbers while it runs: the chain's two rates, the heat's jump and decay, the market speed, and the tail loading that tilts a spot price's shape with the pace (see the market volatility channel note). configure is a command like a pin. It applies at the loop's now and goes in the trace as a configure event. What has passed stays as it was. The heat keeps its level and fades at the new decay. The dwell in progress keeps its unit draw, so it ends at the new rate's share of it. The calm multiplier is computed again, so the mean pace stays at one. The clock refuses a set whose loop gain is one or more, a rate or a speed that is not positive, or rates that leave no positive calm multiplier. The refusal names the number, and then nothing changes.

The clock counts simulated seconds as it counts tau: each stretch of tau times the speed in force while it passed. A tick's simulated seconds are the change in that count, so a speed change reaches every spot feed at its next tick. A curve fixes its step from the speed when it starts, because a scheme that only approximates its law is checked against that step. It keeps that step until it starts again, and it counts a tick's simulated seconds in steps of that length. The settings are on the shell (synthetic regime set) and on the Market regime tab, read only for a tenant.

The loop keeps two records a screen reads. Each heat reading, taken every 100 ms, carries the market's intensity λ(t): the ticks a second every series' law asks for at that instant, its intensity times its sensitivity times the pace. And an arrival history counts the ticks of each group by slot: an hour by the second and ten minutes by the tenth of a second, with the mean pace and the active share of each slot. A series names its group, and a feed names its asset class. The history is a count, not a list of ticks, so it costs the same at any tick rate; the Market screen folds it into a temporal raster, a column a minute and a cell a second of it, where a regime spell shows as a band of columns.

The scheduler holds no database, messaging or logging code, so it keeps the rule of ores.analytics.quant. The synthetic side holds only the feeds. Any other consumer can pace itself the same way without linking the synthetic service.

The note Synthetic market data generation: approach lists the modes a user may choose and the control panel that sets them. This page says where the choice lives in the code.

2.2. The arrival laws

arrival_law in domain/arrival_law.hpp is a variant of three values. Each carries all the state that decides its next gap, in the series' own operational time.

  • periodic_law gives the same gap every time, after a first gap that starts part way in, at a phase it draws. It is the default. A feed that wants no randomness uses it.
  • poisson_law gives an exponential gap (Poisson Process).
  • hawkes_law gives a gap that depends on the earlier ones (Hawkes Process). It starts at its stationary excess, so a run starts at its long-run rate.

draw_gap takes the law as const, and only record_event moves it, so a gap that is drawn and not used leaves the law as it was. make_arrival_law builds the law a spec names and throws on a mode it does not know. A silent fallback to a periodic timer would hide a mistyped mode as a feed that quietly stays regular.

Every draw comes from an rng_stream ( math/rng_stream.hpp ): Philox4x32-10 keyed by the run seed, the series key from the source name, and the purpose. A series' draws do not depend on the thread, the order the feeds were made in, or how many there are. The run seed is the service's --run-seed option, and the service logs it when the run starts.

2.2.1. Why "periodic"

The obvious word, "steady", is taken. std::chrono::steady_clock and the asio steady_timer mean a clock that never goes backwards, which says nothing about how far apart the ticks are. "Fixed" is vague about what is fixed. A point process with equal gaps is called periodic in the literature, and a reader of the code meets the same word. The "Fixed interval" label in the approach note is the periodic arrival law.

2.3. The parameters

A feed names its mode and one set of numbers.

Mode Parameters Meaning
Periodic ticks per hour Equal gaps.
Poisson ticks per hour The mean rate. Gaps are exponential.
Hawkes ticks per hour, branching ratio, decay The long-run mean rate, how clustered the stream is, and how fast a burst fades.

The Hawkes row uses the parametrisation in Hawkes Process: the mean rate \(\bar\lambda\), the branching ratio \(n\) and the decay \(\beta\). The factory turns them into the baseline \(\mu = \bar\lambda(1-n)\) and the jump \(\alpha = n\beta\) inside hawkes_law. A Hawkes law runs in operational time, so its branching ratio is the same in a calm market and a busy one, and a burst fades faster in wall time when the market is busy. ticks_per_hour keeps its meaning in every mode. A person who switches from Periodic to Hawkes keeps the same load on the pipeline, and changes only how that load is spread in time. The approach note's \((\mu, \alpha, \beta)\) is the same process in other coordinates.

The unit of an arrival law is the operational second, and ticks_per_hour is the unit of the config. make_arrival_law converts once. At a pace of one, an operational second is a wall second. The process has no notion of a unit, and a template parameter would add noise for no gain.

2.4. How the elapsed time reaches the price

A gap can be half a tick or three ticks, so a value process cannot assume a step of one fixed length. The step takes the elapsed time:

  • i_stochastic_process has next(double elapsed). elapsed is the time since the last step, counted in nominal ticks, where one nominal tick is the tick length the process is configured for: the length of one period at the feed's ticks_per_hour. It has no unit and it needs no calendar. next() means next(1.0).
  • The feed turns the wall seconds since its last tick into nominal ticks by dividing by the nominal period. A periodic series in a market at a pace of one passes exactly one. A Poisson or Hawkes series at the same ticks_per_hour passes numbers whose long-run mean is one.

A process applies elapsed by the rule that is exact for its law:

Process Step for elapsed \(s\)
Geometric mixture The component's log-return is \(N(m_k s,\ \sigma_k^2 s)\). Each component is a Brownian motion seen for \(s\) ticks.
Arithmetic mixture The same, on the increment.
Ornstein-Uhlenbeck The exact transition with \(\Delta t = s \cdot dt\). The formula holds for any step.

The mixture rule is a modelling choice. Each regime is a Brownian motion, so means scale with \(s\) and variances scale with \(s\), and the weights stay as set. It is exact if the regime holds for the whole step. At \(s = 1\) each rule is one nominal tick, so a periodic feed with a given seed always steps the same sequence.

2.4.1. What the rule gives

The price process, in real time, does not depend on the tick times. A feed that ticks twice as often sees half-size steps, and the sum of those steps over an hour has the same law. Volatility per unit of real time is constant, so a burst of ticks shows many small moves and a lull shows few large ones. A Hawkes clock changes when the price is observed and leaves the price alone. This is also the limit of the model. In a live market bursts and large moves come together, and here they are independent. Linking the two is a different model.

2.4.2. Processes that do not take it

The short-rate processes of the yield curve path precompute their step from dt. A lattice such as Black-Derman-Toy has steps fixed by its construction, and the matrix processes (Heath-Jarrow-Morton, Libor market, the affine and quadratic Gaussian models) precompute a covariance for one step. Only the three engines above take next(elapsed). The others throw when called with an elapsed other than one, and the validation refuses a non-periodic mode for a process type that does not support it. The IR curve feed is periodic, and its discount_factor(ticks_ahead) counts nominal ticks.

2.5. What validation guards

validate_process_parameters and its arrival companions (validate_arrival_process_parameters, validate_step_compatibility) hold the arrival rules. The service applies them when a feed is built; the editors offer only what a process can follow, so the curve screens offer the periodic mode alone.

  • the rate is positive and finite;
  • the decay is positive and finite;
  • the branching ratio is at least zero and strictly below one;
  • a non-periodic mode is allowed only for a process whose supports_variable_step() answers true, which validate_step_compatibility decides;

A branching ratio of one or more gives a stream that does not settle. A ratio very close to one gives gaps so short that the pipeline cannot keep up. The Hawkes process also stops after a fixed number of events in one request and reports that it did so, which guards the tick loop against a parameter set that the rules let through.

2.6. What the pipeline sees

Nothing changes after the feed. A tick carries its time, the marketdata ingest loop plans and stores it, and the rule in No code path may behave differently because a source is simulated holds: the arrival law is a property of the generator and no later stage reads it.

3. See also

Emacs 29.3 (Org mode 9.6.15)