Task: Design and build ACME compute package publishing (client lib, shell, Qt)

Table of Contents

This page documents a task in the Update vendored ORE Engine to 1.8.16.0 story. It captures the goal, current status, acceptance, and any notes or results.

Goal

There is exactly one way to publish a compute app engine package (the seed's per-triplet key shape), implemented once and shared between ores.shell (a new manual publish command, run after ACME core provisioning) and ores.qt (a new standalone Upload Engines dialog, ready to slot into the future Acme wizard). The system is ready to run compute jobs once ACME publishing (core provisioning plus this package publish step) is done.

Status

Field Value
State DONE
Parent story Update vendored ORE Engine to 1.8.16.0
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-07-31

Acceptance

  • A new ores.compute.client component provides a package_publisher usable identically from ores.shell and ores.qt: resolves platform_id, computes local SHA256, uploads to the canonical {app_name}/{version}/{app_name}-{version}-{platform_code}.tar.gz key (the only key shape – the wizard's legacy packages/{app_version_id} shape is retired), verifies the server-returned SHA256, then upserts app/app_version/app_version_platform via the existing save_app_request/save_app_version_request. AppProvisionerWizard is retargeted to this shared publisher with one upload per selected platform. A new ores.shell command performs the same publish non-interactively, reading the binary from a new publish/vendor-packages/ directory populated at build time from external/ore/packages/. A new standalone ores.qt Upload Engines dialog does the same interactively.

Plan

Depends on the sibling task Verify compute grid package SHA256 on download's server-side hashing (handle_put returning sha256) – do that one first.

Discovered while scoping this: AppProvisionerWizard already does most of this end-to-end today (save_app_request=/ =save_app_version_request over NATS, direct QNetworkAccessManager HTTP PUT for the binary) – so this is a retarget and share, not a build-from-scratch. Current gaps this task closes: (a) the wizard uses the "legacy" packages/{app_version_id} key and applies one package_uri to every selected platform (a known, already-commented shortcut in the code); (b) no non-interactive equivalent exists for ores.shell; (c) no SHA256 anywhere in the flow (task 8DBD1BAE).

  1. New ores.compute.client component (mirrors ores.refdata.client=/ =ores.iam.client). package_publisher class:
    • Resolve platform_id from platform_code via the existing list_platforms_request.
    • Compute local SHA256 of the file.
    • Canonical key: compute_storage::package_key(app_name, version, platform_code, ".tar.gz") – the ONLY shape going forward.
    • Upload via ores.storage.client's transfer helper (or an equivalent usable from both a headless shell command and a Qt dialog wanting progress reporting – decide during implementation whether one implementation serves both, or a thin per-caller wrapper around a shared core upload+verify step).
    • Compare the server-returned SHA256 (from the sibling task) against the local one; fail loudly on mismatch.
    • save_app_request (idempotent upsert by name) then save_app_version_request with platforms[{platform_id, package_uri, sha256}]=.
  2. Vendor the binary: new publish/vendor-packages/ directory, CMake-copied from external/ore/packages/ at build time (mirrors the deploy_skills=/=deploy_settings target pattern) – ships with every build, findable via a path relative to the running executable, same trick already used for user_manual.qch.
  3. Retarget AppProvisionerWizard::PackageUploadPage to the shared package_publisher, looping over every selected platform (one upload each) instead of the current single-URI-for-all shortcut.
  4. ores.shell: new manual command performing the same publish non-interactively (app name/version/platform/file as arguments, defaulting the file to publish/vendor-packages/), intended to be run once after ACME's own "core" party/org provisioning completes – not auto-folded into provision party --source acme.
  5. ores.qt: new standalone "Upload Engines" dialog using the same publisher. No Acme-specific wizard exists yet (task F9D8112C, still BACKLOG) to embed this into, so build it reachable standalone for now (e.g. from the Compute Console), designed to slot into that wizard's "select ACME" step once built.

Notes

Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario, run through the QA Validation Runner panel) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
Verify ACME compute package publishing PASSED Ran to completion after fixing 7 real bugs surfaced by the run itself (see * Result).

PRs

PR Title
#1800 [compute] Design and build ACME compute package publishing (client lib, shell, Qt)

Review

# Comment summary File Decision Notes
1 Fetch-then-merge doesn't guard against the fetch itself failing (raised in all 4 review passes) compute_commands.cpp, UploadEnginesDialog.cpp Fixed existing_platforms() now returns std::optional; both callers abort the publish rather than silently proceed with an empty platform list on fetch failure
2 Nit: #include <algorithm> breaks include ordering compute_commands.cpp Fixed Moved next to the other std-library includes
3 SHA256 mismatch leaves a bad object live at the canonical storage key package_publisher.cpp, storage_routes.cpp Declined (for now) Needs a DELETE-on-mismatch or temp-key-rename change to storage_routes; real gap, scoped as follow-up rather than expanding this PR
4 package_publisher::publish() runs synchronously on the Qt UI thread, blocking on large uploads AppProvisionerWizard.cpp, UploadEnginesDialog.cpp Declined (for now) Real UX issue for ~57MB uploads; needs a QtConcurrent::run refactor across both call sites, scoped as follow-up
5 Concurrent-publish race: no optimistic locking between fetch and save compute_commands.cpp, UploadEnginesDialog.cpp Declined Admin-only, low-frequency operation; reviewers themselves concluded acceptable to defer
6 No tests/ directory for the new ores.compute.client component projects/ores.compute/client/ Declined (for now) Covered by the manual QA test scenario recorded above; unit tests for the JSON/mismatch logic are a reasonable follow-up
7 max_body_size (256MB) applies server-wide, not scoped to the storage route http_server_options.hpp, http_session.cpp Declined Already tracked in EA916A96-0F86-42EF-BC26-CB1E7857B3EC (stream large uploads to disk), which subsumes proper route-scoped handling
8 vendor_compute_packages CMake glob is configure-time only / empty-glob risk CMakeLists.txt Declined Repo always ships at least the linux package today; no CONFIGURE_DEPENDS needed for this PR's scope
9 Storage upload happens before DB registration – orphaned blob on save failure compute_commands.cpp, UploadEnginesDialog.cpp Declined Self-healing on retry (deterministic key); low severity

Result

Shipped as designed in * Plan, all five steps:

  1. New ores.compute.client component: package_publisher (upload + client/server SHA256 verification), shared unchanged by both callers below.
  2. publish/vendor-packages/ populated at build time from external/ore/packages/ via a new vendor_compute_packages CMake target.
  3. AppProvisionerWizard::PackageUploadPage retargeted to one upload per selected platform via the shared publisher (was: one URI applied to every platform).
  4. ores.shell's compute publish-package command: non-interactive publish, defaulting the file to publish/vendor-packages/.
  5. New standalone ores.qt UploadEnginesDialog, reachable from Compute Console's Apps tab ("Upload Engines"), for publishing against an app/version/platform that already exists.

Also fixed, all discovered while running the manual test scenario against a real build rather than assumed correct:

  • save_app_version_request replaces every platform row for a version wholesale, so a second publish-package call for a different platform used to silently wipe out the first platform's row. Both the shell command and UploadEnginesDialog now fetch existing platforms first and merge before saving.
  • ores.http.server read every request (including large binary uploads) through Beast's default 1MB body limit, so the ~57MB ORE package upload failed with "Broken pipe". Raised via a new http_server_options::max_body_size (256MB); a follow-up capture (EA916A96-0F86-42EF-BC26-CB1E7857B3EC) tracks the more idiomatic disk-streaming fix as separate work.
  • ComputeConsoleWindow's detail dialogs (App/AppVersion/Batch/ WorkUnit) popped up as independent top-level windows via a local show_detail_as_window hack instead of docking into the main window's MDI area like every other entity dialog in the app. ComputeConsoleWindow now takes mainWindow=/=mdiArea and docks via DetachableMdiSubWindow, matching AppController's convention. AppProvisionerWizard=/=UploadEnginesDialog (both wizard/dialog-type windows) instead get the app's existing Qt::Window wizard convention (same reasoning as SystemProvisionerWizard et al.).
  • AppVersionDetailDialog's platforms/packages silently failed to load for any existing version: the call site set the version (setVersion(), which gates its platform fetch on !createMode_) before setCreateMode(false), so it always ran against the default createMode_ = true=.
  • AppProvisionerWizard's Audit page queried change reasons in the common category, which has zero applies_to_new reasons (amend-only) – should be system, matching the createMode_ ? "system" : "common" convention already used by AppDetailDialog=/=WorkunitDetailDialog.
  • Both new Qt call sites used ClientManager::storedUsername() (the raw login string, e.g. "tenant_admin@acme_corporation") for modified_by=/=performed_by instead of currentUsername() (the resolved bare account username), tripping ores_iam_validate_account_username_fn's check.
  • QaValidationRunnerWidget's screenshot capture always grabbed its own top-level window (the MainWindow), so any window a tester detached out of Compute Console was invisible to a screenshot. Now resolves qApp->activeWindow() at capture time instead.

Filed two backlog captures for follow-up: NATS protocol-message shape normalisation/inventory (5E934799-3E38-4C36-8BE6-7CDCC64DD958), and streaming large HTTP uploads to disk (EA916A96-0F86-42EF-BC26-CB1E7857B3EC).

Emacs 29.3 (Org mode 9.6.15)