The scheduler loop

The scheduler holds one timer and no fixed grid. It sleeps until the earliest next occurrence among the jobs it holds, and at that instant it fires every job whose occurrence has arrived. Nothing else wakes the loop: there is no per-job timer, and a wake that finds a job due is the only wake that has work to do.

Source: https://github.com/OreStudio/OreStudio/blob/main/doc/knowledge/architecture/scheduler_wake.puml. Regenerate with plantuml -tpng doc/knowledge/architecture/scheduler_wake.puml.

The cost of a wake follows the jobs that are due rather than the jobs that are held: only a due job has a handler selected and an action started. A job that is not due costs nothing at a wake, because its next occurrence was computed when it entered the job set or when it last fired.

The loop holds the active jobs and each job's next occurrence. It reads the jobs across all tenants, because it runs the system's jobs as well as a tenant's, and it keeps them for the wakes that follow.

The loop does not poll the job table. A change to a job definition is delivered as an event, and the handler for that event re-reads the job set and re-arms the timer. The re-arm is part of the contract rather than an optimisation: a new or edited job may be due sooner than the instant the loop is sleeping towards, and a loop that re-read the set without re-arming would serve that job late.

Two properties follow from the re-read. An inactive or deleted job leaves the set and stops being an occurrence the loop sleeps towards, and a job the loop already holds keeps its next occurrence, so a change to one job neither resets nor delays another.

At each wake the loop fires every job whose occurrence has arrived. An occurrence has arrived when the job's active flag is true and the next occurrence after its last recorded firing is at or before the current instant, which is the condition the cron expression defines. The loop holds no other notion of when a job should run, which is why changing an expression needs no change to the loop.

For each due job the loop writes the firing, selects the handler for the job's action type, starts the action, and recomputes that job's next occurrence. The loop's part in an action ends at that hand-off.

Three properties follow, and all three are deliberate.

• Jobs do not delay each other. The actions of different jobs run independently, so a job whose action takes a minute does not hold back the jobs fired in the same wake.
• A job can overlap itself. A job whose action takes longer than its recurrence interval can have more than one firing in flight. Each firing has its own record, and the live status read reports the number in flight.
• The loop is independent of the action's result. The result is written to the firing record when the action returns, and it does not affect the next occurrence the loop sleeps towards.

A cron expression names wall-clock instants, and a timer measures elapsed time. The two part company when the wall clock moves: a step from a time source or a daylight-saving change moves the instants a job names while a timer keeps its original duration.

The sleep is therefore bounded and re-derived. The loop never sleeps on the strength of one reading of the clock for longer than the bound, and on waking it derives the next occurrence from the current wall clock rather than from the interval it slept. A clock adjustment can therefore delay a firing by no more than the bound, and it cannot leave the loop asleep past an instant that has already arrived.

At startup the loop reads the active jobs and, for each, the instant of its most recent recorded firing. That instant is the reference point for the due condition and the base of the job's first next occurrence, so a job that already ran is not run again because the process restarted. The reference comes from the firing record: the firing is what the loop reads to decide when a job is next due.

The loop holds jobs, not work. It keeps no queue of pending actions and no backlog of missed firings, so a firing either runs at its occurrence or does not run at all. Two limits follow.

• The loop does not retry. A failed action is recorded as failed, and the job's next execution is the next occurrence its expression defines. A retry would be a second firing, which the expression did not define.
• The loop does not track the work an action starts. An action may start work that outlives it, and the loop neither waits for that work nor reads its result. That work belongs to the component that owns it, and Scheduling and workflows defines the boundary.