Unify the codebase's three separate Beast HTTP client implementations

Table of Contents

This page is a capture in the inbox bucket of the product backlog — a pre-sprint idea, not yet pulled into a sprint as a story.

What

Three independent, hand-written Beast-based HTTP client/request-read implementations exist in the codebase: ores.storage::net::http_client (projects/ores.storage/src/net/http_client.cpp, GET/PUT for object storage), ores.compute.wrapper::net::http_client (projects/ores.compute/wrapper/src/net/http_client.cpp, package download/upload for compute jobs), and ores.http.api's per-connection request-read loop (http_session::run(), which every JSON API route and the storage routes share). Replace all three with one shared Beast-based HTTP client/request-reading component: GET/PUT with http::file_body streaming to/from disk (never a full in-memory buffer), a single sane and configurable body_limit, and used identically by ores.storage, ores.compute.wrapper, and ores.http.api's session loop.

Why

Discovered while fixing the ACME compute package publishing flow (task 0F768B80-184A-44C7-B725-A594A873AB4A): all three independently hit the exact same bug – Beast's default 1MB body_limit rejecting a multi-MB compute engine package – and each had to be found and fixed one at a time as manual testing surfaced it (server-side request read, ores.storage's client GET, then ores.compute.wrapper's own separate download()), rather than being caught once by a single, already-correct shared implementation. ores.compute.wrapper's client additionally buffers whole file uploads into memory as a plain std::string instead of streaming from disk via file_body like ores.storage's version already does – another divergence that wouldn't exist with one shared implementation.

References

  • projects/ores.storage/src/net/http_client.cpp
  • projects/ores.compute/wrapper/src/net/http_client.cpp
  • projects/ores.http/api/src/net/http_session.cpp
  • projects/ores.http/api/include/ores.http.api/net/http_server_options.hpp (max_body_size, added as the interim per-fix patch)

See also

Emacs 29.3 (Org mode 9.6.15)