Scheduled Work Acting as a Tenant

Table of Contents

This page defines the service architecture pattern Scheduled Work Acting as a Tenant in the pattern language. It is a composite: work that starts from a schedule, with no caller, acts as a tenant and a party through a grant, a token exchange and a token cache. In ORE Studio scheduled reports use it, and so does any timer-driven work that reads or writes a party's data.

1. Also known as

  • Run grant and run token, the names the reporting design uses.

2. Context

A person sets up work in a tenant and a party, and a scheduler starts it later, many times. The scheduler and the workflow engine are services, so their identity has no tenant and no party. The work reads and writes data that row-level security scopes by tenant and party. Hundreds or thousands of such runs from different tenants are in flight at once.

3. Problem

How does work that a schedule starts act as the tenant and the party it belongs to, when no caller and no caller's token exists?

4. Forces

  • The service's own identity sees no party data.
  • There is no caller token to relay, and a stored token would be a credential at rest.
  • The person who set up the work must be able to stop it, and must not grant more than they hold.
  • A run can last longer than any token.
  • Thousands of runs must not mean thousands of standing sessions, or an exchange per request.

5. Solution

The composite combines three patterns in a fixed order:

  1. Consent, once: Grant Management. When a person schedules the work, the owning service creates a grant on their behalf with their token. It stores the grant id with the schedule.
  2. Start, at each firing: Client Credentials. The scheduler triggers the owning service as itself. The owning service starts the run with the run id and the grant id, and with no token.
  3. Act, at each step: Token Exchange behind a cache. A step service looks up the run token for the grant id and the run id in its memory cache. On a miss, it exchanges its own token and the grant id for a run token that lasts minutes. It calls the data owners with the run token in Authorization, and they act as the tenant and the party.

The composite is easiest to read as a map: each boundary between two participants uses one pattern, and the start message and the step command carry ids, never a token.

pattern_scheduled_work_acting_as_a_tenant_boundaries.png

Figure 1: Scheduled Work Acting as a Tenant: the pattern each boundary uses, from the person's session to the store that row-level security guards.

Source: [[./pattern_scheduled_work_acting_as_a_tenant_boundaries.puml][pattern_scheduled_work_acting_as_a_tenant_boundaries.puml]]. Regenerate with plantuml -tpng doc/knowledge/service_architecture_patterns/pattern_scheduled_work_acting_as_a_tenant_boundaries.puml.

The combination adds three rules of its own:

  • No token in durable state. The run's messages, rows and logs carry the grant id and the run id, never a token.
  • One cache key per run. The cache key is the grant id and the run id, so a token names exactly one run.
  • Lapse stops the work. A refused exchange with the reason grant_lapsed fails the run and suspends the schedule until a person consents again.

pattern_scheduled_work_acting_as_a_tenant.png

Figure 2: Scheduled Work Acting as a Tenant: a person consents once, the scheduler starts each run as itself, and each step exchanges the grant for a short-lived run token behind a cache.

Source: [[./pattern_scheduled_work_acting_as_a_tenant.puml][pattern_scheduled_work_acting_as_a_tenant.puml]]. Regenerate with plantuml -tpng doc/knowledge/service_architecture_patterns/pattern_scheduled_work_acting_as_a_tenant.puml.

6. Failure handling

Concern What the pattern does
Errors grant_lapsed or a revoked grant fails the run and suspends the schedule. A forbidden from a data owner fails the step and names the permission, which is a defect in the role.
Timeouts An exchange that does not answer is retried with back-off inside the step's budget, then the step fails. The step's deadline bounds the whole wait.
Retries token_expired from a data owner causes one exchange and one retry. A restarted step service exchanges again on its first miss.
Idempotency Every step converges on a repeat, as the workflow engine requires. The exchange itself is safe to repeat.
Observability The run id is in the token, in the correlation id and on every log line. IAM's issue log records every run token. Measure exchanges, cache hits and refusals by reason.

7. Pitfalls

  • The service's identity used for the run. The run reads no party data and reports no error. This is the failure the composite exists to prevent.
  • The scheduling person's token stored. It expires in minutes, and stored it is a credential at rest.
  • A token in the workflow request. The engine persists its request, so the token becomes durable. Carry the grant id instead.
  • No cache. Every owner call becomes an exchange, and IAM carries the load of every run.
  • A cache keyed by grant only. Two runs of one schedule then share a token, and the run id in the token is wrong for one of them.
  • Revocation assumed to be instant. A run in flight keeps working until its current token expires. Size the token's lifetime to the delay the business accepts.

8. Security

The composite inherits the bounds of its parts. The grant limits the scope to a role made for the work and to one party. The exchange is limited to the services in the grant's audience. Each token lasts minutes and names its run. No credential exists at rest. The person who consented, and the service that acted, both appear in every audit row the run writes.

9. Resulting context

Benefits
Scheduled work acts in its tenant and party, and row-level security and the owners' permission checks apply unchanged. The cost per run is a few exchanges, whatever the number of tenants.
Liabilities
IAM holds grants and serves exchanges, so it is on the path of every new run. A person's departure suspends their schedules, so a team must re-consent.

10. In ORE Studio

  • The implementation. Run grants and run tokens is IAM's form of the composite, the same for every component: the grant entity, the exchange checks, the token's claims and the cache rules.
  • Scheduled reports. Report execution at scale applies the composite: the ReportRun role, the audience of reporting, ORE and compute, and the moments reporting creates and revokes a grant.
  • Market data adapters. The identity an adapter runs as is the counterexample: an adapter acts as no tenant and no party, so it needs no grant and no token exchange.
  • The expiry sweep. Approval request expiry applies it: the job belongs to no tenant, its close reads every tenant's overdue requests in one statement, and the notice that follows is each asker's own.
  • Any timer-driven work in a tenant. A component whose scheduled job must read or write party data uses this composite rather than a design of its own.

11. Related patterns

12. Sources

The composite has no single source. Each part cites its own:

13. See also

Emacs 29.3 (Org mode 9.6.15)