The compute wrapper's HTTP download opens its destination before it checks the status

Table of Contents

This page is a capture in the inbox bucket of the product backlog — a pre-sprint idea, not yet pulled into a sprint as a story.

1. What

ores.compute.wrapper carries its own Beast HTTP client, and its download opens the destination file before it looks at the response status:

std::filesystem::create_directories(dest.parent_path());                            // :78
parser.get().body().open(dest.string().c_str(), beast::file_mode::write, open_ec);  // :82
http::read(stream, buf, parser);
if (parser.get().result_int() < 200 || parser.get().result_int() >= 300)            // :89
    throw std::runtime_error(...);

A 404 therefore truncates a file that was already at that path, leaves the error body in a file that was not, and only then throws. ores.storage had the identical shape in http_client::get, and Bring ores.storage to the clean standard fixed it by reading the header first and opening the destination only for a 2xx:

http::read_header(stream, buf, parser);
if (parser.get().result_int() < 200 || parser.get().result_int() >= 300)
    throw std::runtime_error(...);
// only now create the directory and open the body sink

The wrapper's upload is not affected: it reads from disk and checks the status before anything is written.

2. Why

The wrapper's download is how a compute node receives its engine package and its input tarballs, so its destination is usually the artefact the node already has. Destroying the previous artefact on a failed download is worse than failing and leaving it in place, and nothing reports the difference: the caller sees the same exception either way.

The two clients are one behaviour written twice, which is the capture Unify the codebase's three separate Beast HTTP client implementations exists to collapse. That consolidation is the lasting fix and would retire this defect with the file. Until it lands, this is a standalone correctness fix of three moved lines.

3. References

  • projects/ores.compute/wrapper/src/net/http_client.cpp:56-97 — download, opening at :82 and checking at :89.
  • projects/ores.storage/src/net/http_client.cpp — the same method after the fix.
  • projects/ores.storage/tests/http_client_tests.cpp:107-137 — two cases that fail against the old order, one for a destination that did not exist and one for a file that did.

4. See also

Emacs 29.3 (Org mode 9.6.15)