One import run prototype
Table of Contents
| Item | Where |
|---|---|
| Prototype | the prototype |
| Gemini review | none |
| Group | Data exchange prototypes |
| Journeys | See what a failed family left behind, Retry a failed or interrupted import, Reproduce a defect, Import market data from a feed, Seed a fresh environment, Promote an environment |
1. Summary
After the commit there is one question, and it is per family: what was written, what was refused, what was skipped, and what a failure left behind. The journeys ask it about the same attempt record, so this is one screen in different states. It also draws the difference the design decision turns on: one atomic commit, or today's write per family. Above the record sits the workflow instance that performed the import, so the person sees the run while it runs and not only after it ends. Beside it sits the sandbox the run fills, so the same view shows the virtual portfolios and books appearing when the commit completes.
It is throwaway. Open the run.
2. Detail
2.1. What it argues
- A family is a row. The record names every family, including the one that refused and the one that was skipped, and every part that crosses a service.
- The mechanism is visible on the screen. The toggle between "one commit" and "per-family writes" changes what a failure left behind: nothing, or every family written before it.
- The failed family is named. The design draft's reply carries the family and the counts; today's save does not, and the gap is printed where the family name would be.
- A retry is the same import, not a new file. The applied families do not run again, and the retry control sits on the record rather than on a workflow screen the person must find.
- The trade retry still doubles the trades. The seed's second run is drawn with its counts, because no trade is matched by its ORE id today.
- Who may read the run is scoped. The maker, the decider and the same party see it; another party's run is absent.
- The sandbox is part of the record. The attempt names the sandbox it targeted, and the tree below the workflow region shows the sandbox's existing virtual books and the ones the commit creates. A stage step creates no row there; the commit is the step that makes the imported tree live. If the run compensates, the tree stays as it was.
2.2. The workflow run
Progress belongs here and not on another screen, because the run is the attempt: the engine exposes one instance's step list, and a separate progress screen would split the retry from the record it resumes.
- The region is one instance. It reads
workflow.v1.instances.listwithtarget_kind=data_exchange_importandtarget_id=<import id>, thenworkflow.v1.ops.get_workflow_stepsfor that instance. That read answers the run's status and one summary per declared step: name, label, description, status,started_at,completed_at, error and log. - The re-read follows the watch on the workflow component and the instance entity. The state-change notification carries the changed entity, its timestamps and its tenant and not the steps, so the screen reads after it.
- The instance states drawn are
in_progress,completed,failed,compensatingandcompensated. Before the start message there is no instance, and the screen says so instead of drawing a pending run. A step drawspending,in_progress,completed,completed_with_warnings,failedorcompensated. - The deadline is the definition's:
data_step_timeout(900 s) for a step that stages,write_step_timeout(120 s) for a step that writes the record. The engine measures silence from the dispatch, fails a silent step on a pass every 30 s, and stops the run infailedwhatever policy the definition declares. - Retry is
workflow.v1.ops.retry_workflow_instance. It is offered because this definition declaresstop: the run parks infailedand keeps every completed step. A definition that declaredcompensatewould reachcompensatedand offer no retry. - Success with warnings is success. A step that reports
completed_with_warningsadvances the run, is never compensated, and leaves the instancecompleted; its warnings are log entries. The screen's?state=partialmaps to that and not to a partial instance state. - Per-family counts arrive with the step that produced them. A stage step's result carries its family's staged count; the commit's result carries the written counts of every family together, because one transaction produces them together. A per-family bar inside the commit would state something the engine cannot know.
- The region cannot show what only the engine knows: how long a step has been
silent before a pass fires, whether the service that owns the subject is
alive, and whether a late completion is real. It draws
started_atand the budget, and says the pass decides. - The sandbox tree is driven by the same state, not by a second clock. It shows the created portfolios and virtual books only when the commit reached a terminal success state, and it says that a stage step writes nothing into the tree. The sandbox path and id sit beside the instance identity.
2.3. Variants and states
?case=onboard|seed|defect|feed|one-dataset|promote|party,
?state=applied|failed|partial|expired,
?mode=proposed|today, ?fail=<part>, ?attempt=1|2,
?wf=staged|in_progress|waiting|failed|retrying|completed|warnings|compensated,
?play=1,
?target=sb-import-onboard|sb-sample-seed|sb-import-repro|sb-import-feed.
The bar switches the trigger, the outcome, the mechanism, the workflow state
and the attempt; Play walks the workflow states one every 1.6 s. The sandbox
tree follows the attempt record: an applied outcome fills it, a failed or
compensated one leaves it empty, and a completed workflow region against a
failed record says which of the two the tree follows. ?wf=staged is the state with no instance at all: it is what "staged
and awaiting review" means to the engine.
2.4. What is real and what is not
The parts, the family names, the FX skip, the operation subjects and the
atomic-versus-per-family difference are the journeys' and the draft design's.
The state names, the subjects, the budgets, the failure policies and the
progress read are the engine's. The workflow type
data_exchange_import_workflow, its step subjects and the per-family stage
replies are the draft's and are not registered: the engine holds
ore_import_workflow and run_configuration_import_workflow, one step each.
The sandbox rules are the model's: a sandbox is anchored at an official
portfolio, a sandbox root portfolio has no parent, and a virtual book carries
no ledger reference and is not sweepable. The ids, counts, timestamps, sample
names and duplicate fixings are invented, and the whole workflow region is a
script on a clock.
2.5. Screens the workflow connection touches
- Entry: names the type, the policy and the declared steps before anything runs, and says that no such definition exists today.
- Run: follows the one instance, draws the step states, the budget, the retry and the warnings.
- Review: a decider's publish starts a second instance, because an instance
has no way to wait for a person; the staging instance ends
completed. - Register: one row per instance, read by
target_kindandtarget_id, so a party's history is a list of runs rather than a list of files.
2.6. What the workflow docs contradict
- The screen treats a retry as a new attempt. The retry verb resumes the same instance at the failed step under the same step identity, and only a shell re-run starts a new instance. The engine records no attempt number, so "attempt 2" is this screen's label.
- The screen's
partialhas no instance state behind it. The real pair is a failed run with the completed steps kept, orcompleted_with_warningsin acompletedrun. - The screen's
expiredis the staged-row sweep, not a step deadline. A deadline makes a stepfailedand the runfailed; it never rolls back. - The step states note declares
pending, and the progress read's documented status filter does not list it. The screen renderspendingfor every step the engine has not created yet. - A per-family resume cannot exist while an import is one step. The registered import definitions declare a single step, so a retry redoes the whole import; the family-per-step list is the draft's, not the engine's.
2.7. The verdict
Not signed off. The owner must answer the question this screen puts in the middle: is a half-applied ORE file ever acceptable? The trade import says yes by design and the conventions import says no, and the two cannot both be true under one mechanism. The sandbox makes another question visible: when a run fails after the book-tree rows landed, the sandbox holds them and the conventions published to the party do not roll back. Is that a clean failure, or does the sandbox only hide the half-import?
3. See also
- Data exchange prototypes — the group page, and the journeys each screen serves.
- Data exchange journeys — the journeys this screen answers.