NATS entity protocol specification

A subject's first segment is the component that owns it, and the component that owns it is the one that serves it. A subject that no single component owns still belongs to one: a concept shared by every entity is owned by the component whose service answers for it, and there is no bare, ownerless top-level namespace.

This is what keeps request/reply correct. A request subject is subscribed by one component's queue group, so the reply comes from the service that owns the resource. A generic subject that every component subscribes to is answered by whichever service wins the race, which is not a property any caller can rely on.

On a request subject, the last segment is one verb from the closed set. The resource segment is a plural noun in snake_case. list_by_<relation> carries one further word so that a scoped read is still one subject per operation, and a sub-resource's name carries the collection it holds after the resource it belongs to.

A deployment may prepend a prefix to every subject, so that several deployments can share one bus without colliding. The prefix is configuration rather than protocol: subjects are written relative and the prefix is applied when a client connects. A message's contract is stated on the relative form.

Each (resource, verb) pair has exactly one subject. Two components never publish the same subject, and one component never declares the same subject twice. Where a resource is reachable from two directions – a junction whose rows are addressable by either side – the subject is still one, and the direction is a field of the request rather than a second subject.

A collection that belongs to one entity is a sub-resource, and it is a resource like any other: it has a name, a key, its own subjects, and it is read with the standard verbs. It is named for the entity it belongs to, followed by the collection:

{component}.v1.{resource}_{collection}.{verb}

A sub-resource exists because a read that returns something other than a page of the resource is not a read of the resource. The versions of an entity are the case in point: a read that returns version records is not a read of the entity, and giving it a verb of its own would put a second kind of thing behind the verb slot. Naming the collection instead keeps the grammar at four segments, keeps one subject per operation, and lets the collection inherit the list contract rather than restate it.

A sub-resource's key is composite: it is the parent's key together with the member's own identifier. The composite-key rule below already covers it, so a sub-resource needs no machinery that an ordinary composite-key entity does not already need.

A sub-resource may be read-only. Versions are written by the database and not by a caller, so the versions collection declares list and get and no write verb – for the same reason a read-only entity declares none.

list returns one page of entities and the total that the filter matches.

list_<resource>_request {
    offset : uint32          -- default 0
    limit  : uint32          -- default 100
    order  : order           -- the sort, stated explicitly
    filter : <resource>_filter   -- optional; absent means no filter
    as_of  : optional of string  -- optional; the instant to resolve at
}
list_<resource>_response {
    result : result
    <resources> : list of <resource>
    total  : uint64          -- how many the filter matches, ignoring the page
}

Three rules make the contract unambiguous.

• offset, limit and total are unconditional. Every list is paged, every list reports its total, and no caller has to discover whether this particular resource pages. The total is the count before the page is applied, so a caller can drive paging to the end without a second operation. That is what it costs: the count is taken on every list, whether or not the caller wants it, which makes every list two queries rather than one. Uniformity is bought with that and is worth it, because the alternative is a client that cannot know whether the number it holds is the total or the size of the page it happens to have.
• Ordering is stated, never implied. A page of an unordered set is not reproducible, so the request names its sort and the service honours it. Where a caller asks for no particular order, the order is by key, so that paging is stable across calls.
• Filtering is a record, not a query language. A filter is a record with one optional member per filterable field, each carrying the field's own type. There is no free-text query string and no expression to parse. A resource that supports no filtering simply has no filter record.

list replaces every read that returned a filtered set: an entity's versions and the entities beneath a node in a hierarchy are scoped reads of collections in their own right, and they differ from the plain case only in how the scope is named.

A row whose store keeps its validity window can be resolved at an instant rather than at a version number. The list carries the instant in as_of: when it is stated, the read returns the rows whose own window covers it – @c valid_from <= as_of < valid_to – and when it is absent, the read is the ordinary page.

-- absent: one page of the current state
list_<resource>_request { offset, limit, order }
-- stated: every row that was current at that instant, in key order
list_<resource>_request { as_of }

Four things follow from it, and each is deliberate.

• It names an instant, not a version. A versions read names a version number and returns the row that version produced, validity window and all. This read names a point in time and returns what was current then. The two answer different questions and neither replaces the other.
• It is the entity's own window. A composite child's as-of read overlaps a parent version's window; this one resolves the entity's own row. The two are composed by a caller that wants the composite object as it stood.
• The instant is a condition, not a page bound. Every row current at that instant is returned, so offset, limit and total do not narrow it. The reply's total is the size of the set it returned.
• It is declared, not inferred. A resource offers it when the model says so. A model that does not declare it serves the page alone, and the field is absent from its request.

get returns exactly one entity, addressed by its full key.

get_<resource>_request {
    key : <resource>_key
}
get_<resource>_response {
    result : result
    <resource> : optional of <resource>
}

There is no separate "found" flag: an entity that does not exist is the missing outcome. ok always carries an entity, and missing never does.

get is the operation a screen uses to show one record, and it is the reason a screen never has to read a page to find a row in it.

get_many returns the entities named by a set of keys, and answers in the shape of its question.

get_many_<resources>_request {
    keys : list of <resource>_key
}
get_many_<resources>_response {
    result  : result
    entries : list of <resource>_lookup
}
<resource>_lookup {
    key        : <resource>_key
    <resource> : optional of <resource>
}

The reply carries one entry per requested key, in the order the keys were asked for, so the correspondence between a request and its reply is positional and a caller never has to diff the two to find out what it did not get.

An entity that does not exist is an entry whose entity is empty, not a missing entry and not a failure: asking for a set is not a claim that every member of it exists, so the outcome stays ok and the caller reads absence from the entry. Keys may repeat, and each occurrence gets its own entry, because a positional reply is only unambiguous if it answers the question that was asked.

A key set carries at most 1000 keys, the same bound a batch carries (see Batching), so that one request cannot ask for an unbounded reply.

An entity whose rows are versioned has a versions sub-resource, and its versions are read with the verbs every other collection uses.

list_<resource>_versions_request {
    key    : <resource>_key   -- the entity whose versions these are
    offset : uint32
    limit  : uint32
    order  : order
    filter : <resource>_versions_filter
}
list_<resource>_versions_response {
    result   : result
    versions : list of <resource>_version
    total    : uint64
}
get_<resource>_version_request {
    key : <resource>_version_key
}
get_<resource>_version_response {
    result  : result
    version : <resource>_version
}
<resource>_version_key {
    <resource> : <resource>_key
    version    : uint32
}

A <resource>_version is a member of the versions collection: it carries the version number, the audit provenance the service derived when it wrote that version, and the entity's fields as they stood at that version. Because the member type is the collection's own, reading versions is an ordinary read of a resource and nothing about it is a special case.

The collection is read-only. A version is written by the database when the row changes, never by a caller, so list and get are the only verbs it declares.

The version axis is a filter, not a second selection mechanism. A <resource>_versions_filter record carries an exact version and the from_version and to_version bounds, each optional and each typed, alongside whatever else the entity's versions may be filtered by – who modified them, and when they were recorded.

A point-in-time read then falls out of the same shape: a caller that wants the entity as it stood at version 7 lists the versions filtered to version 7, and a caller that wants the last month of changes lists the versions recorded in that window. There is no separate operation for either, and no second way to ask.

The parent's key is one field of the collection's key and also a field of the request, so a composite-key entity's versions are addressed exactly as the entity itself is.

list_by_<relation> is list scoped to the entities related to one other entity, addressed by that entity's key. It carries the relation's key in place of part of the filter, and is otherwise list exactly: same paging, same total, same stated order.

list_by_<relation>_<resources>_request {
    <relation> : <relation>_key
    scope      : direct | subtree   -- default direct
    offset     : uint32
    limit      : uint32
    order      : order
    filter     : <resource>_filter
}
list_by_<relation>_<resources>_response {
    result      : result
    <resources> : list of <resource>
    total       : uint64
}

A relation is named in the subject suffix and its key is a field of the request, and the two always agree. Scoping by a related entity is not a filter, because an authorisation decision may depend on the relation, and a filter is not something a caller's access is evaluated against.

Where the relation is an entity's parent, subtree scopes the page to every entity below the named one rather than to its immediate children. The reply is still a page of entities, each carrying its own parent key as any entity does, so a caller assembles the tree from the keys in the reply: it needs no second read and no node type of its own. A page of a subtree need not be connected, so a caller that renders a whole tree pages to the end of the subtree, which the unconditional total lets it drive.

A client sends a write record: the entity's user-owned fields, and nothing else.

<resource>_write {
    ... the fields the user owns, and no others ...
}

The write record is a distinct type from the entity a read returns. A read returns the entity with its server-owned fields populated, because a caller needs to display and reason about them. A write carries the write record, because a caller has no business asserting them. The two are never the same type, so a client cannot accidentally round-trip a read into a write.

A field is server-owned when the service can derive it, and when a client asserting it would be a way to lie. Server-owned fields never appear in a write record:

A service derives every one of these itself. It does not read them from a request, and it does not trust them if a request carries them anyway: a request that carries a server-owned field is invalid, so that a client is told it is wrong rather than silently ignored. This is the rule that makes tenancy and provenance trustworthy: no caller can write another tenant's data, or claim another person's authorship, because neither is a value the caller supplies.

What a caller does supply about a write, beyond the entity's fields, is its intent: the reason the change is being made, and any commentary the operator recorded. Intent is user-owned data, so it travels in the write record's operation rather than being derived.

change_intent {
    reason_code : string
    commentary  : string   -- optional
}

Intention is required of every write, because a change with no stated reason is not auditable. A service that rejects a write with no intent answers invalid.

A write states what it believes about the entity it is about to change, and the service refuses the write when the belief is false. This is how two writers cannot silently overwrite each other, and how a create cannot silently become an overwrite.

precondition {
    kind    : any | must_not_exist | must_match_version
    version : uint32   -- present only for must_match_version
}

The three are distinct on purpose. A create sends must_not_exist and fails loudly when the key is taken, which is what makes a duplicate create detectable. A caller updating a row it read sends must_match_version with the version it read, and learns that someone else got there first. A caller deliberately replacing a row, or republishing it, sends any. A revert is a write like any other: it sends the fields of the version being restored and states the version it believes is current, so it takes must_match_version and lands as a new version rather than erasing the ones after it. No integer doubles as a mode: there is no value of version that means "do not check", because that is what conflated a create with an overwrite.

A write with no precondition is invalid. Stating the belief is not optional.

A precondition belongs to one change, never to a request that carries several. A batch of updates is a set of rows each at its own version, so a single version claim could not describe it, and a batch-level claim would either weaken the check to any or reject a batch that is entirely valid.

Batching is how a caller changes a set of entities in one operation. It is not a separate family of messages: a batch is the same element repeated, and the single-entity verb is the batch of one.

<resource>_change {
    write        : <resource>_write
    precondition : precondition
}
put_<resource>_request {
    change : <resource>_change
    intent : change_intent
}
put_<resource>_response {
    result     : result
    <resource> : <resource>
}
put_many_<resources>_request {
    changes : list of <resource>_change
    intent  : change_intent
}
put_many_<resources>_response {
    result      : result
    <resources> : list of <resource>
}

A write answers with the entity as it now stands, so that the caller learns the version and the audit provenance the service assigned without a second read. A batch answers with the entities in the order the changes were sent.

Four rules make batching coherent across every entity.

• Every change states its own precondition. The element is where the belief lives, so a batch may create one entity and update another, and each is checked against what the caller actually believes about it. A verb's single form and its batch form share one element type, so they cannot drift apart in what they permit.
• The batch is atomic; the elements are independent in what they claim. The set lands together or not at all. A precondition that fails anywhere makes the whole operation conflict, and nothing is written – which is what makes a batch safe to retry, because a half-applied batch is not a state a caller has to reason about.
• Intent is stated once for the set. A batch is one act by one operator, so the reason for it is one reason. Per-element intent would let one operation tell several different stories about why it happened.
• A batch has a bound. A batch carries at most 1000 elements and a page carries at most 1000 entities, with a list's limit defaulting to 100. The bounds are part of the protocol rather than a runtime surprise, so a caller can size its chunks and a generated message can enforce them. A request that exceeds a bound is invalid.

A batch is not a way to escape authorisation. Every element is authorised as if it had been sent alone, and the operation is denied if any element is not permitted, so a batch cannot reach a row its author would not have reached singly.

A delete names the key of what it removes and carries its own precondition, on the same terms as a write: removing a row a caller has not read is a claim that it wants to remove whatever is there. The reason the deletion is recorded travels as change intent.

<resource>_removal {
    key          : <resource>_key
    precondition : precondition
}
delete_<resource>_request {
    removal : <resource>_removal
    intent  : change_intent
}
delete_many_<resources>_request {
    removals : list of <resource>_removal
    intent   : change_intent
}

The element carries the precondition for the reason a write's element does: a batch of deletions is a set of rows at different versions, and a caller removing several rows it read is claiming something about each one. Both responses carry a result and nothing else, because a deletion has no data to return.

A removal states any or must_match_version; must_not_exist describes a create and has no meaning for a removal, so a removal that states it is invalid. A row that does not exist is missing and a row that exists at another version is conflict, so a caller can tell "someone else removed it" from "someone else changed it". Under any both are ok: a removal with nothing left to remove has still achieved what it asked for, which is what makes a retried delete safe.

The batch is one transaction, and it does not report per-entity success: the whole operation either happened or it did not.

A caller is authenticated before any operation is reached, and its identity is carried by the connection in the envelope, never in a payload. Every operation runs against that identity. A payload cannot assert one.

Tenant and party are derived from the authenticated token and applied by the service. They are not request parameters and not filter fields. A caller therefore cannot read or write outside its own tenancy by asking to, and the rule that enforces this is the same rule that keeps server-owned fields off the wire – there is nothing to override because there is no field to send.

Operations that act outside the caller's own tenancy are not expressed by widening a request. They are addressed by a subject in a reserved administrative namespace, authorised separately and more narrowly than the ordinary subject, and audited as privileged. This keeps the common path free of an override and makes every privileged operation a visible address rather than an argument.

Authorisation is per operation and per resource. The service checks it before performing the operation, and answers denied when it fails. A verb that modifies state is never authorised by the same permission as a verb that reads, so that read access cannot be widened into write access by a request that names a different verb.

Authorisation is not the client's concern: a caller does not ask whether it may do something, it does it and is told. A capability a client displays is a hint, not a grant.

Envelope headers are the registry for everything that is not business data.

A service propagates the correlation id onto every message it publishes while serving a request, so that one cause is traceable across components.

A command that cannot be expressed idempotently at the data layer carries an idempotency key, and a service that sees a key it has already served returns the original result rather than performing the operation twice. This is what makes a retry safe on a bus that delivers at least once.

Authorization, X-Delegated-Authorization and X-Error are handled at the transport boundary, before a payload is decoded, so that a failure to authenticate is reported even though no operation was reached. X-Error carries one of a small closed set of codes on a reply with an empty body, and covers only those failures: everything an operation itself decides is reported by the result inside the response.