集中式日志聚合

概念概述

集中式日志聚合（Centralised Log Aggregation）指的是：集中式日志聚合把各实例的日志汇入一个可检索存储，用可查询、可关联的结构化记录取代逐主机文件。

从更严格的定义看，Log aggregation is the collection, shipping, and indexing of log records from many sources into a central system so they can be searched, correlated, and retained beyond a single host. 该概念在工程实践中的价值，在于它能帮助团队用统一的语言描述一个具体的质量属性或设计约束，从而在评审与协作中减少歧义。

为什么需要它

It is essential once a service runs on more than one instance, because a request may touch hosts the operator cannot predict. 它与 The USE Method for Resource Analysis、Symptom-Based Alerting、Trace Context Propagation 等概念经常一起出现，用于解决相近的工程问题。理解它的适用场景，有助于在多种备选方案之间做出有依据的取舍，而不是凭直觉选择技术方案。

常见误解

• logging everything verbatim improves debuggability
• centralised logs can store every field without cost concerns

需要避免的反模式

• shipping raw request payloads including personal data
• raising a volume that makes the index expensive and slow to query

实践要点

1. Structured fields such as request identifiers, tenant, and latency make logs queryable, whereas free-form messages force fragile text searching.
2. Volume drives both cost and query latency, so sample verbose records and drop noise rather than indexing everything by default.
3. Logs must respect privacy constraints: redact secrets and personal data at the source, before shipping, because a central store widens access.

参考来源

https://opentelemetry.io/docs/concepts/signals/logs/

代码示例

package main

import (
	"log/slog"
	"os"
)

func main() {
	logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
		Level: slog.LevelInfo,
	}))

	logger.Info("request handled",
		slog.String("method", "GET"),
		slog.String("path", "/users/42"),
		slog.Int("status", 200),
		slog.Duration("latency", 0),
	)

	logger.Error("dependency failed",
		slog.String("dependency", "postgres"),
		slog.Int("attempt", 3),
	)
}

#!/usr/bin/env bash
set -euo pipefail

LOG_LEVEL="${LOG_LEVEL:-info}"

declare -A LEVELS=([debug]=0 [info]=1 [warn]=2 [error]=3)

log() {
  local level="${1:?level required}"
  shift
  local threshold="${LEVELS[${LOG_LEVEL}]:-1}"
  local current="${LEVELS[${level}]:-1}"
  if (( current < threshold )); then
    return 0
  fi
  printf '{"timestamp":"%s","level":"%s","message":"%s"}\n' \
    "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "${level}" "$*" >&2
}

log info "job started"
log debug "this line is suppressed at info level"
log error "dependency unavailable"