Queue-Based Load Leveling
Table of Contents
This page defines the service architecture pattern
Queue-Based Load Leveling in the pattern language.
A queue sits between the work that arrives and the service that does it, so a
peak waits instead of overloading the service. In ORE Studio report runs over
their quota wait as queued, and compute holds assignments in JetStream.
1. Also known as
- Load levelling.
2. Context
Work arrives in bursts: thousands of reports fire at the same minute. The service that does the work has a fixed capacity.
3. Problem
How does a service survive a burst of work larger than its capacity?
4. Forces
- Capacity sized for the peak is idle most of the day.
- Refusing the burst loses work.
- A person must be able to see work that waits.
- Waiting work must not be lost on a restart.
5. Solution
Put a durable queue between the source and the service. The source adds work at any rate; the service takes it at its own rate. The queue is visible, so a person sees what waits, and durable, so a restart loses nothing.
Figure 1: Queue-Based Load Leveling: a burst of firings starts up to the quota and waits as queued for the rest.
Source: [[./pattern_queue_based_load_leveling.puml][pattern_queue_based_load_leveling.puml]]. Regenerate with
plantuml -tpng doc/knowledge/service_architecture_patterns/pattern_queue_based_load_leveling.puml.
6. Failure handling
| Concern | What the pattern does |
|---|---|
| Errors | A failed item leaves the queue with its error; it does not block the items behind it. |
| Timeouts | Time spent waiting counts against nothing until the item starts. |
| Retries | An item taken but not finished returns to the queue. |
| Idempotency | An item may be taken twice; see Idempotent Receiver. |
| Observability | Queue length and wait time are measured, per tenant. |
7. Pitfalls
- A queue nobody can see. Work waits with no state a person can read.
- A queue that only grows. Capacity below the average rate is not levelling; the backlog never clears.
- Plain arrival order across tenants. One tenant's burst delays everyone; see Fair Share.
8. Security
No security consequence beyond availability.
9. Resulting context
- Benefits
- Bursts become waits, and the service runs at a steady rate.
- Liabilities
- Latency during bursts, and a queue to watch.
10. In ORE Studio
- Report execution at scale — a run over its quota is born
queuedand promoted as runs finish. - Compute job lifecycle — JetStream holds each assignment until a node acknowledges it.
- How the compute grid queues and claims work — the compute queue is the table of results, and a node claims a job when it is free.
11. Related patterns
- Fair Share — the order the queue serves.
- Competing Consumers — the members that drain it.
- Throttling — the alternative when work cannot wait.
12. NATS patterns
The NATS service patterns hub states how each of these is written in
ores.nats code:
- Durable Subscription — the durable consumer that holds work for an absent service.
- Stream Provisioning at Start — the stream that has to exist before the queue can hold anything.
- Acknowledge What Was Handled — how the reader drains the queue without losing work.
13. Sources
- Microsoft, Azure Architecture Center, Cloud design patterns, Queue-Based Load Leveling pattern.
14. See also
- Service architecture patterns — the structure note that orders this pattern language, and where to read this page in it.