Propagate the Correlation Id

• NATS: the Nats-Correlation-Id header.
• The pattern language: Correlation Identifier.

One top-level action fans out across services: a request is served, the handler calls another service, that service writes a row whose trigger publishes an event, and a fourth service reacts. Each hop is in a different log file with nothing in common but a timestamp.

How does a person join the log lines of one request when no process can see the whole request?

• No transaction spans the services, and no shared memory exists.
• A request may be served by any replica of each service.
• The trace key must exist from the first hop, before any service has seen the request.
• The key must not be confused with a message's own identity, which serves a different purpose.

One key per top-level request, carried in a header, generated at the entry point if absent and forwarded by every hop:

// At the entry of every handler: read the key, or make one.
[[maybe_unused]] const auto cid = log_handler_entry(lg(), msg);

// On every outbound call the handler makes.
return session.with_correlation_id(cid)
    .authenticated_request(subject, payload);

Three rules:

• Generate once, at the entry point. A key present in the headers is reused; a missing one is created. A hop must never replace a key it was given.
• Forward on every outbound call. A single hop that drops the key breaks the trail at exactly the place it is needed.
• Do not reuse it for anything else. It is distinct from a message's deduplication identity; one identifies a request, the other a message.

The session id is a second, coarser key: one per login session rather than one per request, so all activity of a user session can be grouped. It travels the same way and is recorded on the same receipt line.

• Generating a new key at every hop. Each service's lines are their own trail and none of them join.
• Not forwarding the key. The trail stops at the first call the handler makes.
• A key only on the success path. Failures are the case that needs it most.
• Confusing the two keys. A session id is not a request key, and a message identity is neither.

Benefits

One grep on the key returns every line every service wrote for one request, in order.

Liabilities

The key must be threaded through every call the handler makes, which is easy to forget at a new call site.

`extract_or_generate_correlation_id` reads or creates the key; `nats_client::with_correlation_id` forwards it, as `with_session_id` does for the session key. The engine forwards the key on every step command with the step and instance identity as sibling headers.

• Announce Every Handled Message — the line the key is written on.
• Distributed Tracing in the pattern language — the fuller form this is the first step of.

• microservices.io, Distributed tracing — a trace id propagated across service calls.
• W3C, Trace Context — the standard header for carrying a trace identity.
• NATS documentation, Headers.

• NATS service patterns — the structure note that orders this pattern language, and where to read this page in it.