Throttling
Table of Contents
This page defines the service architecture pattern Throttling in the pattern language. A service limits how much one caller or tenant may ask for in a period, so no single source can exhaust it. In ORE Studio IAM limits token exchanges per service and per tenant, and reporting caps the runs of each tenant and party.
1. Also known as
- Rate limiting.
- Quota, for a limit on concurrent work rather than a rate.
2. Context
A shared service serves many callers. One caller, by fault or by load, can send more than the service can serve.
3. Problem
How does a service protect itself, and its other callers, from one source of excess load?
4. Forces
- Every caller deserves service; none may take all of it.
- Some work can wait; some must be refused.
- A refused caller needs to know when to try again.
5. Solution
Count each caller's use over a period, or its work in flight, against a limit. Over the limit, delay the request, queue it, or refuse it with a reason and a time to try again. Log every refusal with the caller and the limit.
Figure 1: Throttling: a request within its caller's and tenant's limits is served; over them it is queued or refused.
Source: [[./pattern_throttling.puml][pattern_throttling.puml]]. Regenerate with
plantuml -tpng doc/knowledge/service_architecture_patterns/pattern_throttling.puml.
6. Failure handling
| Concern | What the pattern does |
|---|---|
| Errors | A refusal names the limit and, where it can, when to try again. |
| Timeouts | Does not apply: the limit counts use, not time. |
| Retries | A refused caller retries after the stated time, with back-off; see Retry. |
| Idempotency | Does not apply. |
| Observability | Refusals are counted per caller, per tenant and per limit. |
7. Pitfalls
- Limits nobody sees until they bite. State them, and measure use against them.
- A limit per replica. Several replicas each grant the limit, so the real limit multiplies.
- Retries that ignore the refusal's time. They arrive at once and are refused again.
8. Security
Availability: a faulty or hostile caller cannot exhaust the service. Limits per tenant keep one tenant's fault from reaching another.
9. Resulting context
- Benefits
- The service stays up under excess load, and the cause is named.
- Liabilities
- Legitimate bursts are delayed or refused, and limits need tuning.
10. In ORE Studio
- Run grants and run tokens — IAM limits exchanges per service and per tenant.
- Report execution at scale — quotas cap the running runs of each tenant and party.
- Delivery threads enqueue and a bounded pool runs the handlers — a full queue answers busy or drops by policy, so a burst costs refusals, never memory.
11. Related patterns
- Queue-Based Load Leveling — the alternative when work can wait.
- Bulkhead — isolation behind the limit.
- Fair Share — dividing what the limit admits.
12. Sources
- Microsoft, Azure Architecture Center, Cloud design patterns, Throttling pattern and Rate Limiting pattern.
- Betsy Beyer et al., Site Reliability Engineering, O'Reilly, 2016, chapter 21, "Handling Overload".
13. See also
- Service architecture patterns — the structure note that orders this pattern language, and where to read this page in it.