Task: Mint a credential for long automated work

Table of Contents

This page documents a task in the Fix the tenor basis swap convention model and the document TU — the remaining work story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

A long-running automated operation holds a credential built for it: its own lifetime, the tenant, party and actor fixed, and renewal on refusal, so a provisioning run is watched to completion instead of dying on a token that was never meant to outlive a picker.

2. Status

Field Value
State DONE
Parent story Fix the tenor basis swap convention model and the document TU — the remaining work
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-27

3. Acceptance

  • A token states the lifetime it actually carries, or party selection gets its own response field, and the client computes its refresh from that number.
  • Automated work uses its own credential, not the interactive party-selection token.
  • A refused call with token_expired renews the credential and retries while a wait is in progress.
  • ACME provisioning reaches active with no manual re-login.

4. Plan

4.1. What is already right, and what is missing

service_token_provider (projects/ores.iam/client/src/client/service_token_provider.cpp) already does the right thing: it reads access_lifetime_s from the login response, computes a refresh margin of a fifth of it, and re-mints. For a party-selected login that response carries 300 s, so a client using the provider refreshes at 240 s and never expires.

The shell does not use it. projects/ores.shell/application/src/app/application.cpp holds the session token from auto_login statically, and no code in projects/ores.shell mentions session_expired_error at all, so the refusal from nats_client.cpp:153 travels up to the command and aborts it. Provisioning, driven by the shell, therefore outlives its own token and dies at the poll.

4.2. The order of work

  1. Give the shell a renewable session: hold a token provider beside the nats_client, so every request renews at the margin the server stated.
  2. Handle session_expired_error where renewal is impossible: re-login once and retry the operation, as internal_request_client already does for services.
  3. Only if a party-selected session still needs its own lifetime, give party selection its own response field rather than overloading access_lifetime_s, so a client can never read one number as two.
  4. Prove it with the same run: ACME provisioning watched to active with no manual re-login, then the convention page smoke that waited behind it.

4.3. What was built

The plan above assumed the shell needed a token provider like a service has. It does not: a service knows its account's password and can re-mint a token on a timer, while a shell is a person's session. What it needs is the other half of the client's own contract – the client throws session_expired_error and the caller signs in again. Nothing in the shell did that.

So the renewal moved to the one place that holds a token and can be tested:

  • nats_client takes a session_renewer. On a refused token it renews once and sends the request again, and throws only if the renewal fails. A refusal is safe to retry because the server rejects an expired token before it acts on a request.
  • The shell remembers the credentials that opened the session – from the start-up login or from the login command – and hands the client a way to open a fresh session with them, party selection included.
  • select_party and switch_party now report the lifetime they actually minted rather than leaving a field at a value written into the type, so a client that computes a refresh from it computes from the truth.

The party-selection token keeps its five minutes. It is a step token spent between the login and the choice, and lengthening it was never the fix: the run that was supposed to prove otherwise died waiting on a service that was not running, and its access token expired while it waited.

(Implementation strategy. Written when work starts; key decisions are distilled into the parent story's * Decisions at close, but the plan itself stays — it is the historical record of what we did.)

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
       

6. What the first watched run showed

The run that was meant to prove the token fix instead corrected the diagnosis. It never reached a token problem first.

  • 18:21:20 UTC — the provisioning workflow starts step 0 (system_provision) and reaches synthetic.market_data_providers: in_progress at 18:21:52.
  • No further step line follows for ninety minutes. The step publishes to the synthetic service through NATS and waits for a reply; the service had been dead since 17:09, so the reply could not come.
  • 18:51:31 UTC — a call is refused with token_expired, exactly 1800 s after the 18:21:16 login. That is the party-selected access token expiring, not the 300 s party-selection step token, which was already spent by then.

So the party-selection lifetime was never what killed the run. Two separate defects did, and both are worth fixing:

  1. The wait is bounded, but far too loosely to help. A step command is dispatched through a durable stream, so a consumer that is not running looks exactly like one that is slow: the command is stored and waits. step_publish_timeout is 7200 s, so a provisioning run whose service died waits two hours to say so, and the shell driving it gives up long before that. A dispatch that cannot tell a dead consumer from a slow one should check that the consumer exists, or bound the wait in the unit that owns the step rather than the one watching the run.
  2. The shell holds one token and has no renewal or re-login path, so a run that legitimately takes longer than its access token dies even when every step is healthy.

The synthetic service itself crashed with free(): invalid pointer in ~fx_spot_generation_config, because the publish tree was left holding two builds: 24 of 71 libraries in publish/lib were at 0.0.28 from an earlier date while their .core and .api partners were at 0.0.29. An interrupted link at 10:19 that day stopped the build at step 311 of

  1. The service library was the older build, so its destructor read the

newer struct at the wrong offsets. compass services status now reports this, because nothing did.

7. Notes

8. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

9. PRs

PR Title
#3996 [agile] Close the credential task
#3960 [iam] Keep a long automated run signed in

10. Review

Comment summary File Decision Notes
       

11. Result

Done. A refused token no longer ends a command.

The client renews once on a refusal and sends the request again; the shell gives it the credentials that opened the session, from either sign-in path; party selection reports the lifetime it minted. The check that proves it: with the access lifetime set to one second, a burst of 120 commands was answered in full and IAM logged three sign-ins – one at the start and two renewals. Nothing about the run depended on the person watching it.

Two other defects came out of the same session and are fixed here:

  • The workflow deadline counted a step's time in a queue as silence, so a healthy provisioning step was failed for waiting behind a password change. Silence is now measured from the worker's claim, and a step nobody has claimed is not failed while its service is reporting.
  • compass services status now reports a publish tree that holds two builds, which is what made the synthetic service crash in a destructor earlier the same day.

The convention record page, which was the reason the run was being watched, also failed and is fixed: it read the grouped central row as if it were flat and drew both leg forms' terms on a one-form convention. It is now covered by a harness check against a live fleet.

Not fixed, and captured in the backlog: a provisioning run that a real failure strands still cannot be resumed without knowing its instance id, and a step's command still travels through a durable stream that cannot tell a dead consumer from a slow one.

Emacs 29.3 (Org mode 9.6.15)