Transporting a package

A bundle is a classification, not a file format and not a large message. It is sanctioned by the Bundle classification when its families are related and one owner holds them all. The families are open in number: a structure with its trades, its instruments and its legs is one bundle over trading's tables, and the trading owner applies it in one unit_of_work.

The test is the sentence above. Complete it without naming a file, a screen, a workflow step or a boundary format:

These families may not be written apart because __ must hold at one instant, or because one operation must apply all of them or none.

• If the blank needs a file, a screen, a step or a format to be true, the families are not related. The set is a package's contents, and each family is its own bundle. Split it.
• If the invariant covers only some of the families, the set is two or more bundles. Split it at the invariant.
• If the invariant covers every family and one owner holds them, the set is a bundle, and one transaction applies it.
• If the families are related only by provenance and several owners hold them, no bundle spans them. The set is a package carrying one bundle per owner, and this page is its transport.

A file's contents are never an invariant. A document lists conventions because the format lists them; that is a fact about ORE XML, not about conventions. An export that must be converted to a boundary format is a package when its bundles came from one system at one instant; the conversion needs every part at one instant, and the stated as_of is what gives it that.

The design constraint is the heterogeneity, not the size. Size makes the transport worth designing; heterogeneity across owners is why the transport cannot name the types in one message, and why the atomic unit is the bundle rather than the package.

A package carries a manifest, its parts and an archive, and the bundles it carries are records in the manifest. Each artefact has one writer, one place to live, and a fixed set of readers.

The archive is assembled from staging objects written by the owners. Each owner encodes the parts of its bundle, compresses each entry with zstd, writes the compressed entry to a staging object, and returns a part record with the checksum and both sizes. The producer writes the manifest, then asks object storage to copy each staging entry into the archive under its entry name, at ZIP method 93, without recompressing it, and to add the manifest as the manifest.msgpack entry. It never decodes an entry. The producer deletes the staging objects once the archive's part checksums have been verified. No component ever encodes or compresses another owner's rows.

The package_id is SHA-256 over the canonical manifest bytes with every part.uri set to the empty string, encoded by the same codec in declaration order. The URI is the only field blanked, because it is derived from the id. The digest therefore covers as_of, the target, the producer, the bundles, every part checksum and the compressed layout. A re-encoding changes the id and therefore the key; the part checksums do not change, so semantic identity survives at the part level. A key is never reused for different semantic content.

The manifest and the part references are kilobytes. The data is in object storage. Efficiency comes from that structure, not from the encoding: encoding a package into a message would still move the same bytes onto the bus, and would add a decode step before any owner could start.

The reason a package does not travel as a message is validation, routing, per-bundle retry and the prohibition on an encoded value in a message field — not size.

• Validation. A message is validated before it is accepted. A part's rows cannot be validated until they are decoded against the owner's model, which is the owner's job at its own boundary.
• Routing. One message has one subject and one handler. A package has one bundle per owner and one owner per bundle, and the coordinator must dispatch each bundle to the owner that may write it.
• Per-bundle retry. A failed part must cost one bundle, not a whole package. A message that carries the archive forces a re-send of every bundle to retry one.
• The encoded-value rule. A message field that holds an encoded value has no static type at either end; see Polymorphic payloads over NATS.

A message also has a practical bound, and a package does not. That is a limit on the message, not the reason for the design.

The manifest is a generated, modelled operation type. It is declared in ores.codegen.operation, generated with a schema and a codec, and encoded with the platform wire codec like any other message. It is never free-form JSON, never a std::string field holding JSON, and never inferred from the payload or from the archive. It is versioned, so a reader refuses a layout it does not know instead of guessing. It is the document that states which bundles the package carries, and the parts of each.

The target is not decoration. An apply package with an empty target.sandbox is refused by the coordinator. A package aimed at a sandbox applies inside that sandbox's isolation; the anchor decides who may open the sandbox and who may then see the result, per Sandbox.

bundles is ordered by owner dependency: every bundle is contiguous, and the bundles appear in dependency order. The grouping is what lets one message per bundle preserve the global application order. A manifest that interleaves two bundles of one owner is refused, because a bundle is one atomic unit and no message can apply half of it before the other half.

bundles is O(number of bundles), and each bundle's parts are O(number of parts), never O(number of rows). A part record holds a row_count, not the rows. That bound is what makes the manifest validatable before any bytes move: its size is set by the number of owners, bundles and kinds, not by the size of the portfolio, so a portfolio of a hundred million rows still has a manifest of kilobytes. A manifest that grew with rows would be the message the package exists to avoid, and the coordinator would have to read the data to validate its own routing table.

The archive is a ZIP with one zstd-compressed entry per (owner, kind), named {owner}/{kind}, using ZIP compression method 93, plus the embedded manifest.msgpack entry. The ZIP central directory is the index: it gives every entry's name, compression method, compressed size and offset. The archive is content-addressed: a key is never reused for a different semantic content. A put replaces the whole object; a different compression is a new package_id. A tar_zstd container is allowed only when it carries a seekable zstd frame index that gives each entry's byte range; a plain tar.zst has no index, cannot give random access, and is refused.

Random access is the decisive point. A container with an index gives random access to a single part. An owner seeks to its entry, reads that entry and nothing else. A single large msgpack document cannot do this, because a msgpack array has no index: to reach the owner's part, a reader must decode every row of every earlier part. That would force one owner to read and parse bytes another owner produced, which is exactly what Database per Service forbids. The index is therefore a boundary control, not a performance detail.

The row encoding follows these rules:

1. One entry, one type, one stream. An entry is a stream of msgpack maps, one map per row, and every map is the owner's generated type for that kind. The entry does not hold one array of all rows.
2. Named fields, never positional arrays. A row is a map with string keys. A later field addition adds a key and does not shift an existing value, so an archived package still decodes after the type gains a field. A positional array binds meaning to position; adding a field in the middle silently reinterprets every archived row.
3. Row-by-row streaming decode with bounded memory. The owner decompresses the entry and decodes one map at a time, validates it and writes it inside its transaction. Memory is the size of one row plus the transaction state, not the size of the package. This is what makes a portfolio spanning every instrument type tractable: the number of kinds changes the number of parts, not the memory an owner needs to apply one of them.
4. The part format version is independent of the package format version. part.format_version gates the row encoding; package_format_version gates only the manifest layout and the container rules. A version 1 manifest may contain a version 2 part, and the owner selects the decoder from the part record. A new row field or a new part encoding does not force a new manifest layout.

Every part is checksummed uncompressed: SHA-256 over the concatenated msgpack maps exactly as the owner will decode them. The checksum is therefore a statement about the data, not about the compressor. A later zstd version, a different compression level or a re-encoding of the archive leaves the checksum unchanged, so the part's semantic identity survives a compressor upgrade. The compressed size is recorded separately in part.compressed_size; it describes the object, not the data. A re-encoding produces a new package_id, because the compressed layout is part of the manifest, and the recomputed checksum still matches.

The owner decompresses the entry, hashes the uncompressed stream, and compares the digest with the manifest before it applies the first row. Verification is a first pass over the entry and decoding is a second pass, so memory stays bounded and no row is applied before the digest matches. A mismatch is a failure of the part, never a warning.

Compress the archive, not the messages. The wire codec already gzips a message payload above 4096 bytes, so compressing a manifest or a part-reference message again is compressing compressed data. Do not wrap the archive in gzip either: its entries are already zstd-compressed. Compression belongs to the object-storage layer, where the archive and its entries live.

The coordinator is the apply operation's front end. It reads only the manifest. It validates the version, the target, the required fields, that every bundle names a known owner, that every (owner, kind) pair is unique, and that the bundle order is the declared sequence. It may HEAD the archive object to confirm that it exists; it does not GET, decompress or decode a part, and it does not compute a part checksum. That check belongs to the owner, which reads the bytes anyway.

The coordinator must never parse another owner's data, or database per service is broken. It has no schema for another service's rows, so any parse it performs is a second model that can only drift. The same rule is recorded in No service reads or writes another component's tables and in Database per Service.

It dispatches one message per bundle, carrying the package id, the bundle id, the correlation id, the archive URI, the manifest's as_of and target, and that bundle's part records. A reference, never the data. When a bundle holds several parts, the one message carries all of that bundle's records; the message is still O(parts this owner owns), and the owner applies them in manifest order and reports each part separately. No message carries rows, a document or an archive. The coordinator never merges two bundles into one message, because that would make one message carry two atomic units and two owners' rows.

The part message is a request with a deadline, per Bounded Request. The reply names the part and reports applied, rejected or failed, with the reason and the row count. The bundle's reply is the set of its parts' outcomes.

Source: package_transport_sequence.puml. Regenerate with plantuml -tpng doc/knowledge/architecture/package_transport_sequence.puml.

Source: package_transport_layout.puml. Regenerate with plantuml -tpng doc/knowledge/architecture/package_transport_layout.puml.

The owner's step is the only step that writes rows, and it writes only its own. It receives one message per bundle:

1. Fetch only its bundle's parts, by entry name or byte range, from the archive the message names. It does not fetch the whole archive.
2. Decompress each entry and verify the SHA-256 of the uncompressed stream against part.checksum.
3. Look up the idempotency key (package id, bundle id, kind, checksum). If this part already committed, reply applied with its recorded counts and stop.
4. Compare as_of with the owner's heads. A stale bundle is refused, as Progress, retry, and staleness states.
5. Decode the stream with the codec part.format_version names, one row at a time, and validate each row against the owner's own model: required fields, keys, and references the owner holds.
6. Write each validated row through the generated repository for its kind, inside one unit_of_work for the bundle, in manifest order. Never write a kind's table with a hand-written statement in a part handler; see Entity-table writes go through the repositories.
7. Commit only after the last row, and only if the decoded count equals part.row_count for every part. A row that fails validation, a count that differs and any write error roll back the whole bundle. The bundle is all or nothing.
8. Make every read of the bundle with the unit of work's context, so it runs on the bundle's transaction and sees the bundle's own writes; see A repository works inside an open transaction and never holds credentials.
9. Stamp the session's tenant and party, never a value from the payload.
10. Reply with the counts per part and, on failure, the reason.

The transaction is the owner's, so the bundle is the unit of atomicity. A failure halfway rolls back that bundle alone. No other owner sees a partial write, and no other owner is asked to compensate. A package with several bundles is therefore never atomic as a whole.

The guarantee a package carries depends on how many owners its bundles span. One case is a package that carries a single bundle, and that bundle is written in one transaction. The other is a package whose bundles span several services, and no single transaction covers it. Which case a package is decides the mechanism, and the mechanism decides the guarantee. A package is never atomic across bundles or owners in either case.

Progress and retry are per bundle and per part, never per archive:

• Progress. The coordinator records each part as pending, dispatched, applied, rejected, failed or timed_out, with its row count and reason, and reports each bundle as the set of its parts. The package's progress is the set of its bundles, not a percentage of a blob.
• Retry. A retry re-sends the bundle reference. The archive's content is immutable, so the retry reads the same uncompressed bytes. The apply is idempotent on the key (package id, bundle id, kind, checksum): the owner records that key in its own part-apply ledger in the same transaction as the rows, and a second delivery of the same part finds the key and replies applied without writing again. The coordinator does not re-upload the archive and does not restart the whole package to fix one bundle.
• Rejection. A rejected bundle is a finding about the data, not a transport failure. It is reported with the row or key that failed, and the other bundles are not rolled back.
• Restart. Restarting the coordinator re-offers the outstanding bundles only.

as_of makes a stale package detectable rather than last-writer-wins. It is the bitemporal instant the package is a view of, per Time and Timestamps. A bundle is stale when the owner's heads have moved since that instant: any row the bundle would write has a version whose transaction time is later than as_of, or a key the bundle references in this owner's database was deleted or replaced after as_of. A bundle whose as_of is later than or equal to every relevant change is not stale, and it applies.

The owner refuses a stale bundle. It does not rebase it. A rebase is an explicit new export at a newer as_of, which produces a new manifest and a new package_id; the producer mints it, and no owner edits the manifest or silently applies an older bundle against newer rows.

Refusal is the chosen behaviour. A silent rebase would make each bundle a different instant, which destroys the one property the package exists to provide. The archived package is the evidence of what was imported and when; a rebase that changed its meaning would destroy that evidence. And a refusal is detectable: the user sees the stale bundle and can ask for a fresh package, while a rebase reports success and leaves the mixture invisible.

A portfolio spans currencies, calendars, conventions, curves, trades and every instrument type the book holds. The export must give the ORE XML converter one instant, or the document mixes two books. The artefact is a package; the atomic units inside it are one bundle per owner.

The bundles and their parts. One bundle per owner, and one part per (owner, kind), in application order:

• the refdata bundle: refdata/currency, refdata/calendar, refdata/day_count and refdata/convention;
• the marketdata bundle: marketdata/curve and marketdata/vol_surface;
• the trading bundle: trading/portfolio, trading/book, trading/trade and trading/instrument/*, one part per instrument kind.

This is exactly the case a closed message type cannot carry: the last group gains a member whenever the book gains an instrument kind.

The manifest in outline.

package_format_version  1
container               zip_zstd
producer                {trading-export, 1.0.0, production}
as_of                   2026-10-17T09:00:00.000000Z
target                  {sandbox: empty (export only),
                         anchor: desk-42, party: portfolio-owner}
bundles                 refdata, marketdata, trading (in that order)
                        one part record per (owner, kind) in its bundle

The sequence.

1. The converter fixes as_of and asks the export operation for the portfolio. No owner reads outside that instant.
2. Each owner reads its rows in one transaction, encodes and zstd-compresses its entries, writes a staging object per part, and returns a part record with the checksum, row count and both sizes.
3. The producer writes the manifest, then asks storage to assemble the staging entries into one archive and to add the manifest entry. It verifies every part checksum.
4. The converter reads the manifest, then streams each entry into the ORE XML writer. It writes to a temporary file and renames it only after every part has been read and checked.

The atomicity actually obtained. There is no global transaction. Each owner's bundle is one consistent read at as_of, and the conversion is all or nothing at its own boundary: if any part is missing or fails its checksum, the converter writes no ORE XML at all. A failure does not roll back a read, because there is nothing to roll back; it withholds the output. That is consistency by the instant plus an atomic output file, not distributed atomicity, and not package-wide atomicity.

A developer needs a production defect reproduced from one snapshot of the environment: identity, reference data, market data and trades together. The snapshot is a package carrying one bundle per owner.

The bundles and their parts. The iam bundle: iam/tenant and iam/party. The refdata bundle: refdata/currency, refdata/calendar and refdata/convention. The marketdata bundle: marketdata/curve. The trading bundle: trading/book, trading/portfolio, trading/trade and trading/instrument/*. Owners: ores.iam, ores.refdata, ores.marketdata and ores.trading.

The bundles are not related to one another: iam rows and marketdata rows share no invariant and no operation. They are in one package because one export produced them, which is provenance and not relatedness. The package is therefore never atomic as a whole.

The manifest in outline.

package_format_version  1
container               zip_zstd
producer                {environment-export, 1.0.0, production}
as_of                   2026-10-17T09:00:00.000000Z
target                  {sandbox: 7f3c..., anchor: desk-42,
                         party: 8a21...}
bundles                 iam, refdata, marketdata, trading
                        ordered by reference, not alphabetical

The sequence.

1. The source environment exports the package and stores the archive and manifest, as in the previous example.
2. The development coordinator reads the manifest, checks that the sandbox is open, that its anchor matches target.anchor, and that every bundle names a known owner.
3. It sends one message per bundle in bundles order: ores.iam first, then ores.refdata, then ores.marketdata, then ores.trading.
4. Each owner fetches only its bundle's entries, verifies the checksums, streams the rows, validates them, and applies them in one unit_of_work per bundle.
5. A person examines the sandbox. Promotion to live data is a separate operation with its own review.

The atomicity actually obtained. Each bundle is atomic in its owner's database. If trading rejects a trade after refdata applied, the refdata rows stay in the sandbox and the trading bundle applied nothing; the sandbox is then discarded as a whole, and no official row was touched. The package does not provide global rollback, and the sandbox is what makes that limit safe.

• An opaque archive inside a message field. That is the artefact anti-pattern in Data classifications: the receiver must unpack before it can validate, no owner can be routed to, and one failed row costs the whole archive. It also breaks the encoded-value rule in Polymorphic payloads over NATS.
• A union of unrelated entities presented as one struct. That is the anti-type in Data classifications: one typed array per kind, named after a file or a screen, with no manifest, no owner boundary and no instant.
• A payload the coordinator parses. A coordinator that decodes another owner's rows is a second model. It is the composite scope the classification page refuses, in Data classifications, and it breaks Database per Service. The manifest is the only thing the coordinator reads.
• A package treated as one atomic set. The package is the carriage, not the unit of consistency. A caller that needs all-or-nothing is asking for a bundle, and a bundle belongs to one owner. A package that promises all-or-nothing across owners promises what no transaction can give.
• A substitute for a bundle when one owner suffices. If one owner writes every family and one transaction can hold the invariant, the set is a bundle; use ores.codegen.family for its families, not this transport. The classification page owns that decision, in Data classifications, and the checklist records it, in the data classification checklist.