Log Aggregation
Table of Contents
This page defines the service architecture pattern Log Aggregation in the pattern language. Every service's logs are collected where they can be searched together. In ORE Studio the services log to the console, systemd keeps it in the journal, and the telemetry log store holds the records the logs screen reads.
1. Also known as
- Centralised logging.
2. Context
Many services on several hosts each write their own logs. A failure's cause and its symptom are in different services.
3. Problem
How does a person search the logs of every service at once?
4. Forces
- Logs on each host are read one host at a time.
- Files written by each service need rotation and collection.
- A search across services needs a common key.
5. Solution
Each service writes to standard output, and the host's supervisor collects it. Records also go to a central store with their time, level, source and identifiers. A person searches the store, and a Correlation Identifier joins the lines of one request.
Figure 1: Log Aggregation: services log to the console and to the telemetry log store, and an operator searches both.
Source: [[./pattern_log_aggregation.puml][pattern_log_aggregation.puml]]. Regenerate with
plantuml -tpng doc/knowledge/service_architecture_patterns/pattern_log_aggregation.puml.
6. Failure handling
| Concern | What the pattern does |
|---|---|
| Errors | An ingest that fails does not fail the service that logged. |
| Timeouts | Does not apply. |
| Retries | Ingest retries; a record may be delivered twice. |
| Idempotency | Duplicate records are tolerated, or keyed by their source and time. |
| Observability | This is the pattern's purpose. |
7. Pitfalls
- No common key. The lines of one request cannot be joined; see Correlation Identifier.
- Secrets in log lines. A central store spreads them; never log tokens or passwords.
- A log file as a readiness signal. Removing the file then breaks the start-up; the journal story removed this dependency.
8. Security
The store holds every service's lines, so reading it is a privileged act: the logs screen is for the system administrator. Lines must not carry credentials.
9. Resulting context
- Benefits
- One search across services, and one place to read a failure's whole story.
- Liabilities
- A store to run, and volume to retain.
10. In ORE Studio
- Read the telemetry logs — the operator filters the telemetry log store by time, level, source and session.
11. Related patterns
- Correlation Identifier — the key that joins lines.
- Distributed Tracing — adds the call tree.
12. NATS patterns
The NATS service patterns hub states how each of these is written in
ores.nats code:
- Announce Every Handled Message — the one receipt line per handled message.
- Propagate the Correlation Id — the key that joins one request across services.
- Observe the Connection State — the connection transitions that explain a gap in a service's log.
13. Sources
- Chris Richardson, Pattern: Log aggregation, microservices.io.
- Sam Newman, Building Microservices, second edition, O'Reilly, 2021, chapter 10, "From Monitoring to Observability".
14. See also
- Service architecture patterns — the structure note that orders this pattern language, and where to read this page in it.