Correlation Identifier

• Correlation id, request id, and the X-Request-Id or X-Correlation-Id header of HTTP services.
• Hohpe and Woolf define the pattern for matching a reply to its request. The microservice literature widens it to every hop an action causes, which is the use here.

One action of a person, or one firing of a schedule, becomes requests to several services, workflow steps that run minutes later, and events that other services consume. Each service writes its own log. A failure surfaces in one log, and its cause is in another.

How does a person find every log line, in every service, that one action caused?

• Each service logs on its own; nothing joins the lines of one action.
• Work outlives the request: a workflow step runs after a restart, an event is consumed later.
• A trail with one missing hop is broken from that hop on.
• An identifier from outside can carry anything, including text that forges a log line.
• The identifier must cost nothing and decide nothing.

The origin of an action creates the identifier: the client, the HTTP edge, or a scheduler firing. Every message the action causes carries it. A handler reads it on entry, opens a log scope so that every line it writes names it, forwards it on every outbound call and every event, stores it with any work that outlives the request, and echoes it on the reply.

Source: [[./pattern_correlation_identifier.puml][pattern_correlation_identifier.puml]]. Regenerate with plantuml -tpng doc/knowledge/service_architecture_patterns/pattern_correlation_identifier.puml.

A handler follows a fixed order on entry. It accepts a well-formed id, replaces a malformed one, creates one when none arrived, and says in the log that it did, so a broken trail is visible.

Source: [[./pattern_correlation_identifier_handler.puml][pattern_correlation_identifier_handler.puml]]. Regenerate with plantuml -tpng doc/knowledge/service_architecture_patterns/pattern_correlation_identifier_handler.puml.

• A hop that does not forward. The next service creates a new id, and the trail breaks there. Every hop that does not forward is a gap, and a handler that creates an id must log that it did.

Source: [[./pattern_correlation_identifier_broken_trail.puml][pattern_correlation_identifier_broken_trail.puml]]. Regenerate with plantuml -tpng doc/knowledge/service_architecture_patterns/pattern_correlation_identifier_broken_trail.puml.

• The id on the entry line only. A handler that logs the id once leaves its other lines unjoined. Put the id in a log scope, so every line carries it.
• Work that outlives the request. A workflow step that runs after a restart has no inbound header unless the run stored the id. Store it with the work.
• The id used as an idempotency key. Two different requests of one action share it, so deduplicating on it drops real work.
• An outside id written to the log as it came. A value with a newline forges a log line. Accept only the expected shape.
• A session id used as a correlation id. A session spans many actions; a search for it returns all of them.

The id grants nothing and identifies no one, so a forged id can only mislead a search. The cost of a hostile value is log injection, which the shape check prevents. The id may be stored, unlike a token, because it is not a credential.

Benefits

One search of the logs returns everything one action caused, across services and across time. A person can quote the id from an error.

Liabilities

Every hop must take part, and one that does not breaks the trail. The id shows that lines belong together, not their order in a call tree or how long each hop took.

• The mechanism. ores.nats defines the Nats-Correlation-Id header and extract_or_generate_correlation_id, and nats_client::with_correlation_id forwards it. log_handler_entry in ores.service logs it on entry. The extractor does not yet check the shape: it accepts any non-empty value, so the log-injection pitfall below is open until the story closes it.
• The workflow engine. The start message carries a correlation id, and the engine's events carry it. Report execution relies on it.
• Scheduled reports. Report execution at scale carries the firing's id through every step of a run.
• The rest of the platform. The handlers, the codegen templates, the origins and the log format do not yet carry it through. Follow a request across services from its logs is the story that does, and it states the measured starting point.

• Distributed Tracing — adds the call tree and the timings; the correlation id becomes the trace id.
• Token Relay — travels the same hops, with the same rule: forward what the inbound request carried.
• Scheduled Work Acting as a Tenant — the firing is the origin of a scheduled run, and the run id is in its token.

The NATS service patterns hub states how each of these is written in ores.nats code:

• Propagate the Correlation Id — the Nats-Correlation-Id header, generated once and forwarded by every hop.
• Announce Every Handled Message — the receipt line the key is written on.

• Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns, Addison-Wesley, 2003, Correlation Identifier.
• Sam Newman, Building Microservices, second edition, O'Reilly, 2021, chapter 10, "From Monitoring to Observability" — correlation IDs.
• Chris Richardson, Pattern: Log aggregation and Pattern: Distributed tracing, microservices.io — the request id each log line carries.

• Service architecture patterns — the structure note that orders this pattern language, and where to read this page in it.