Retry
Table of Contents
This page defines the service architecture pattern Retry in the pattern language. An operation that failed for a transient reason is tried again, with a limit and a growing delay, and only when repeating it is safe. In ORE Studio the workflow retry verb resumes a failed run, and an expired token is renewed and retried once.
1. Also known as
- Retry with exponential back-off.
- Resume, for a workflow run restarted at its failed step.
2. Context
A call fails: a timeout, an unavailable service, an expired token. Some failures pass on their own; others will fail the same way every time.
3. Problem
When should a failed operation be tried again, and how often?
4. Forces
- A transient failure passes; a permanent one never does.
- Retries multiply load on a service that is already struggling.
- Repeating a write that already took effect corrupts data.
- Callers that retry together arrive together.
5. Solution
Classify the failure first. Retry only a transient one, and only an operation that is safe to repeat. Wait between attempts, growing the delay and adding jitter so that callers spread out, and stop at a limit or at the caller's budget. A token that expired is renewed before the retry. A run stopped by a failure resumes at its failed step, keeping every completed step.
Figure 1: Retry: only a transient failure of a repeatable operation is retried, with back-off and jitter, within a limit.
Source: [[./pattern_retry.puml][pattern_retry.puml]]. Regenerate with
plantuml -tpng doc/knowledge/service_architecture_patterns/pattern_retry.puml.
6. Failure handling
| Concern | What the pattern does |
|---|---|
| Errors | A permanent error, forbidden or bad_request, fails at once. |
| Timeouts | Each attempt has its own timeout, and all attempts share the caller's budget. |
| Retries | This is the pattern itself. |
| Idempotency | The operation must be idempotent; see Idempotent Receiver. |
| Observability | Each attempt is logged with its number and the reason. |
7. Pitfalls
- Retrying a refusal.
forbiddenwill not change by asking again. - Retries at every layer. Three layers of three attempts make 27 calls.
- No jitter. Clients that failed together retry together.
- Retrying an expired token with the same token. Renew first; the first version of internal actor impersonation retried until its wait timed out.
8. Security
No security consequence, provided a refusal is never retried with a different identity to get round it.
9. Resulting context
- Benefits
- Transient failures heal without a person.
- Liabilities
- Added load and latency, and a need for idempotent operations.
10. In ORE Studio
- Anatomy of a Service — the service client retries only
token_expired, once, after a forced token refresh. - Failure, compensation and retry — the retry verb resumes a failed run at its failed step, under the same step identity.
- Job actions — the scheduler deliberately does not retry a failed action; the next occurrence is the next firing.
- How the compute grid queues and claims work — a lease that ends returns the job to the queue a bounded number of times.
- A store error says whether a retry can succeed — the store error's class says whether a retry can succeed, and a refused write is never retried.
11. Related patterns
- Timeout — the failure most often retried.
- Idempotent Receiver — what makes a retry safe.
- Token Exchange — renew, then retry once.
12. NATS patterns
The NATS service patterns hub states how each of these is written in
ores.nats code:
- Bounded Request — what the caller waits on before it retries.
- Classify Transport Failure — the two failures that call for different retry timing.
- Idempotent Message Identity — what makes a re-sent request safe to send twice.
- Commit Through the Stream — a retried send that must be confirmed as stored.
13. Sources
- Microsoft, Azure Architecture Center, Cloud design patterns, Retry pattern.
- Marc Brooker, Timeouts, retries, and backoff with jitter, the Amazon Builders' Library.
- Betsy Beyer et al., Site Reliability Engineering, O'Reilly, 2016, chapter 22, "Addressing Cascading Failures".
14. See also
- Service architecture patterns — the structure note that orders this pattern language, and where to read this page in it.