Task: Design and build ACME compute package publishing (client lib, shell, Qt)
Table of Contents
This page documents a task in the Update vendored ORE Engine to 1.8.16.0 story. It captures the goal, current status, acceptance, and any notes or results.
Goal
There is exactly one way to publish a compute app engine package (the seed's per-triplet key shape), implemented once and shared between ores.shell (a new manual publish command, run after ACME core provisioning) and ores.qt (a new standalone Upload Engines dialog, ready to slot into the future Acme wizard). The system is ready to run compute jobs once ACME publishing (core provisioning plus this package publish step) is done.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Update vendored ORE Engine to 1.8.16.0 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-07-31 |
Acceptance
- A new ores.compute.client component provides a package_publisher usable identically from ores.shell and ores.qt: resolves platform_id, computes local SHA256, uploads to the canonical {app_name}/{version}/{app_name}-{version}-{platform_code}.tar.gz key (the only key shape – the wizard's legacy packages/{app_version_id} shape is retired), verifies the server-returned SHA256, then upserts app/app_version/app_version_platform via the existing save_app_request/save_app_version_request. AppProvisionerWizard is retargeted to this shared publisher with one upload per selected platform. A new ores.shell command performs the same publish non-interactively, reading the binary from a new publish/vendor-packages/ directory populated at build time from external/ore/packages/. A new standalone ores.qt Upload Engines dialog does the same interactively.
Plan
Depends on the sibling task
Verify compute grid package SHA256 on download's server-side
hashing (handle_put returning sha256) – do that one first.
Discovered while scoping this: AppProvisionerWizard already does
most of this end-to-end today (save_app_request=/
=save_app_version_request over NATS, direct QNetworkAccessManager
HTTP PUT for the binary) – so this is a retarget and share, not a
build-from-scratch. Current gaps this task closes: (a) the wizard
uses the "legacy" packages/{app_version_id} key and applies one
package_uri to every selected platform (a known, already-commented
shortcut in the code); (b) no non-interactive equivalent exists for
ores.shell; (c) no SHA256 anywhere in the flow (task 8DBD1BAE).
- New
ores.compute.clientcomponent (mirrorsores.refdata.client=/ =ores.iam.client).package_publisherclass:- Resolve
platform_idfromplatform_codevia the existinglist_platforms_request. - Compute local SHA256 of the file.
- Canonical key:
compute_storage::package_key(app_name, version, platform_code, ".tar.gz")– the ONLY shape going forward. - Upload via
ores.storage.client's transfer helper (or an equivalent usable from both a headless shell command and a Qt dialog wanting progress reporting – decide during implementation whether one implementation serves both, or a thin per-caller wrapper around a shared core upload+verify step). - Compare the server-returned SHA256 (from the sibling task) against the local one; fail loudly on mismatch.
save_app_request(idempotent upsert by name) thensave_app_version_requestwithplatforms[{platform_id, package_uri, sha256}]=.
- Resolve
- Vendor the binary: new
publish/vendor-packages/directory, CMake-copied fromexternal/ore/packages/at build time (mirrors thedeploy_skills=/=deploy_settingstarget pattern) – ships with every build, findable via a path relative to the running executable, same trick already used foruser_manual.qch. - Retarget
AppProvisionerWizard::PackageUploadPageto the sharedpackage_publisher, looping over every selected platform (one upload each) instead of the current single-URI-for-all shortcut. ores.shell: new manual command performing the same publish non-interactively (app name/version/platform/file as arguments, defaulting the file topublish/vendor-packages/), intended to be run once after ACME's own "core" party/org provisioning completes – not auto-folded intoprovision party --source acme.ores.qt: new standalone "Upload Engines" dialog using the same publisher. No Acme-specific wizard exists yet (task F9D8112C, still BACKLOG) to embed this into, so build it reachable standalone for now (e.g. from the Compute Console), designed to slot into that wizard's "select ACME" step once built.
Notes
Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario, run
through the QA Validation Runner panel) that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
| Verify ACME compute package publishing | PASSED | Ran to completion after fixing 7 real bugs surfaced by the run itself (see * Result). |
PRs
| PR | Title |
|---|---|
| #1800 | [compute] Design and build ACME compute package publishing (client lib, shell, Qt) |
Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | Fetch-then-merge doesn't guard against the fetch itself failing (raised in all 4 review passes) | compute_commands.cpp, UploadEnginesDialog.cpp | Fixed | existing_platforms() now returns std::optional; both callers abort the publish rather than silently proceed with an empty platform list on fetch failure |
| 2 | Nit: #include <algorithm> breaks include ordering | compute_commands.cpp | Fixed | Moved next to the other std-library includes |
| 3 | SHA256 mismatch leaves a bad object live at the canonical storage key | package_publisher.cpp, storage_routes.cpp | Declined (for now) | Needs a DELETE-on-mismatch or temp-key-rename change to storage_routes; real gap, scoped as follow-up rather than expanding this PR |
| 4 | package_publisher::publish() runs synchronously on the Qt UI thread, blocking on large uploads | AppProvisionerWizard.cpp, UploadEnginesDialog.cpp | Declined (for now) | Real UX issue for ~57MB uploads; needs a QtConcurrent::run refactor across both call sites, scoped as follow-up |
| 5 | Concurrent-publish race: no optimistic locking between fetch and save | compute_commands.cpp, UploadEnginesDialog.cpp | Declined | Admin-only, low-frequency operation; reviewers themselves concluded acceptable to defer |
| 6 | No tests/ directory for the new ores.compute.client component | projects/ores.compute/client/ | Declined (for now) | Covered by the manual QA test scenario recorded above; unit tests for the JSON/mismatch logic are a reasonable follow-up |
| 7 | max_body_size (256MB) applies server-wide, not scoped to the storage route | http_server_options.hpp, http_session.cpp | Declined | Already tracked in EA916A96-0F86-42EF-BC26-CB1E7857B3EC (stream large uploads to disk), which subsumes proper route-scoped handling |
| 8 | vendor_compute_packages CMake glob is configure-time only / empty-glob risk | CMakeLists.txt | Declined | Repo always ships at least the linux package today; no CONFIGURE_DEPENDS needed for this PR's scope |
| 9 | Storage upload happens before DB registration – orphaned blob on save failure | compute_commands.cpp, UploadEnginesDialog.cpp | Declined | Self-healing on retry (deterministic key); low severity |
Result
Shipped as designed in * Plan, all five steps:
- New
ores.compute.clientcomponent:package_publisher(upload + client/server SHA256 verification), shared unchanged by both callers below. publish/vendor-packages/populated at build time fromexternal/ore/packages/via a newvendor_compute_packagesCMake target.AppProvisionerWizard::PackageUploadPageretargeted to one upload per selected platform via the shared publisher (was: one URI applied to every platform).ores.shell'scompute publish-packagecommand: non-interactive publish, defaulting the file topublish/vendor-packages/.- New standalone
ores.qtUploadEnginesDialog, reachable from Compute Console's Apps tab ("Upload Engines"), for publishing against an app/version/platform that already exists.
Also fixed, all discovered while running the manual test scenario against a real build rather than assumed correct:
save_app_version_requestreplaces every platform row for a version wholesale, so a secondpublish-packagecall for a different platform used to silently wipe out the first platform's row. Both the shell command andUploadEnginesDialognow fetch existing platforms first and merge before saving.ores.http.serverread every request (including large binary uploads) through Beast's default 1MB body limit, so the ~57MB ORE package upload failed with "Broken pipe". Raised via a newhttp_server_options::max_body_size(256MB); a follow-up capture (EA916A96-0F86-42EF-BC26-CB1E7857B3EC) tracks the more idiomatic disk-streaming fix as separate work.ComputeConsoleWindow's detail dialogs (App/AppVersion/Batch/ WorkUnit) popped up as independent top-level windows via a localshow_detail_as_windowhack instead of docking into the main window's MDI area like every other entity dialog in the app.ComputeConsoleWindownow takesmainWindow=/=mdiAreaand docks viaDetachableMdiSubWindow, matchingAppController's convention.AppProvisionerWizard=/=UploadEnginesDialog(both wizard/dialog-type windows) instead get the app's existingQt::Windowwizard convention (same reasoning asSystemProvisionerWizardet al.).AppVersionDetailDialog's platforms/packages silently failed to load for any existing version: the call site set the version (setVersion(), which gates its platform fetch on!createMode_) beforesetCreateMode(false), so it always ran against the defaultcreateMode_ =true=.AppProvisionerWizard's Audit page queried change reasons in thecommoncategory, which has zeroapplies_to_newreasons (amend-only) – should besystem, matching thecreateMode_ ? "system" : "common"convention already used byAppDetailDialog=/=WorkunitDetailDialog.- Both new Qt call sites used
ClientManager::storedUsername()(the raw login string, e.g. "tenant_admin@acme_corporation") formodified_by=/=performed_byinstead ofcurrentUsername()(the resolved bare account username), trippingores_iam_validate_account_username_fn's check. QaValidationRunnerWidget's screenshot capture always grabbed its own top-level window (the MainWindow), so any window a tester detached out of Compute Console was invisible to a screenshot. Now resolvesqApp->activeWindow()at capture time instead.
Filed two backlog captures for follow-up: NATS protocol-message shape normalisation/inventory (5E934799-3E38-4C36-8BE6-7CDCC64DD958), and streaming large HTTP uploads to disk (EA916A96-0F86-42EF-BC26-CB1E7857B3EC).