Task: Finish IAM per-service pilot: JWT signing failure and HEALTHCHECK

Reproduced IAM standalone in its per-service container (fresh build, docker/stage-runtime.sh --service ores.iam.service), passing NATS/DB config via explicit CLI args (no args_template row for ores.iam.service in this environment's DB, so the controller's own default template's substitutions had to be reconstructed by hand) and ORES_IAM_SERVICE_JWT_PRIVATE_KEY via docker/.env plus the --env "$(cat build/keys/iam-rsa-private.pem)" override PR #1779 added to docker/run-pod.sh.

Result: the JWT key parsed correctly and IAM reached "Service ready." – no "bio read failed" error, confirming this was not a distinct bug needing its own fix. It's the exact same escaped-\n-in-an-env- file corruption already root-caused and fixed generically in PR #1779 (docker/generate-env.sh's quote-stripping plus the --env passthrough for this one variable); the per-service pilot's session simply predated that fix. Confirmed by first reproducing the original failure with the key passed only via a plain --env-file pointing at docker/.env's escaped-\n value (fails, "Invalid tenant ID" was actually a red herring encountered first – see below – but with the --env override alone, in isolation, IAM signs and starts cleanly).

Along the way, hit and fixed two more standalone-specific setup issues, neither a real bug in shipped code:

• --tenant system (the tenant code) is rejected by ores.iam.service's own CLI parsing with "Invalid UUID string" – this binary's --tenant only accepts a UUID directly, unlike some other services' tenant resolution. Needed the literal system-tenant UUID (ffffffff-ffff-ffff-ffff-ffffffffffff) instead. Not a code bug – just something the controller's own args_template building handles for the real fleet, that this manual standalone run had to replicate by hand.
• The Newton-specific storage EIO finding (PR #1779) reproduces locally too, on this box's own filesystem, once --userns=keep-id remaps the container's uid – confirms that finding was never Newton-specific, just uid-remap-dependent (a mode-777 dir baked into the image's own overlay layer, written to as a different uid than built it, hits the same EIO here as on Newton). Host bind-mounting /app/log~/~/app/run instead (same fix as PR #1779) resolves it.

The design task's acceptance ("podman logs shows IAM's own output directly … confirming no –log-to-console plumbing is needed") was untested and turned out wrong: every service logs to a file only (--log-enabled, no console output) by convention, so podman logs captures nothing at all without --log-to-console also being passed – podman's log driver only captures stdout/stderr, and none of our services write there unless told to. Verified: with --log-to-console added, podman logs shows the exact same output as the log file, live. Not a blocker for the HEALTHCHECK itself (which reads the log file directly, not podman logs), but worth recording as the corrected understanding for future per-service rollout: each service's container should pass --log-to-console if operators are expected to use podman logs day-to-day.

cgr.dev/chainguard/glibc-dynamic has no shell – no grep, no test, nothing to script a log-tail check with, and Docker/podman's HEALTHCHECK CMD exec form runs the given program directly rather than through a shell regardless. Wrote a small standalone C program (docker/healthcheck.c), statically linked in its own debian healthcheck-build stage (so it carries no runtime dependency on the final image's glibc at all), that scans every *.log file under /app/log for the literal string "Service ready." and exits 0/1 accordingly – the same marker every service logs via ores.service's *_runner_impl.hpp once it's registered its NATS handlers, i.e. the exact string wait_for_log_ready already polls for natively. Copied into /app/bin/healthcheck in the final stage, wired via HEALTHCHECK --interval=2s --timeout=2s --start-period=60s --retries=3 CMD ["./healthcheck"].

One gotcha: podman silently ignores HEALTHCHECK entirely (with only a build-time warning, easy to miss) unless the image is built with --format docker – the default OCI format has no health-check concept. Confirmed via podman ps showing (healthy) only once rebuilt with that flag.

Verified end-to-end: IAM standalone (no controller exec'ing it) reaches Up ... (healthy) per podman ps, confirmed via podman inspect --format '{{json .State.Health}}' showing 5+ consecutive passing health-check runs.

This is deliberately a pragmatic first cut, not the final design – reading a log file from outside the process is fragile (couples every container's health probe to log wording staying stable) and duplicates state the service already holds. Filed as a follow-up capture rather than building it now, since this task's job was to prove HEALTHCHECK works at all, not to design the ideal probe: Replace log-grep HEALTHCHECK with an in-process NATS health endpoint.