构建变更数据捕获管道

项目概述

本项目实现一个面向 change data capture 的完整系统，核心要解决的问题是 streaming row changes from a transaction log so downstream systems stay in sync without polling。整体由 Log tail reader、Change decoder、Schema projector、Sink writer、Offset tracker 五个模块组成，每个模块单一职责、可独立测试，最后通过集成测试验证端到端正确性。设计上把 a schema change in the source that the projector does not know about and that corrupts the sink 作为第一类需求先行处理，而非事后再补。

架构与关键决策

数据模型为 a stream of change events read from a transaction log, projected against a schema and written to a sink with an offset checkpoint。Log tail reader 定义其余阶段共同依赖的数据形状并在边界处校验输入；Change decoder 承载 change data capture 的正确性逻辑；Schema projector 专门处理 a schema change in the source that the projector does not know about and that corrupts the sink，在失败路径上给出明确的拒绝、重试或降级策略；Sink writer 提供可观测性，每次操作输出带关联 id 的结构化事件；Offset tracker 在过载时按既定方式降级，避免过载演变为数据损坏。配置与代码分离，使风险变更可经 diff 审阅。

实施与测试

按五个阶段依次实现 Log tail reader、Change decoder、Schema projector、Sink writer、Offset tracker，每阶段配套聚焦单测。集成测试组合各模块并驱动真实 change data capture 场景：先验证正常路径，再刻意触发 a schema change in the source that the projector does not know about and that corrupts the sink 并断言系统降级而非损坏。所有依赖均在进程内或被桩替换，用例在 CI 中确定可复现。

限制与后续

• 单区域设计，多区域复制超出范围。
• Log tail reader 默认进程内，未水平共享。
• 无鉴权层，假定部署在强制身份的边缘之后。

代码示例

# setup for change-data-capture
pnpm init
pnpm add pg redis
pnpm add -D typescript tsx vitest

change_data_capture/
├── src/index.ts
├── src/log-tail-reader.ts
├── src/change-decoder.ts
├── tests/change_data_capture.test.ts
└── package.json

// Log tail reader: change data capture core
export function logtailreader(input: LogtailreaderInput): LogtailreaderResult {
  // Build the log tail reader that anchors change data capture and defines the core data shape.
  return processLogtailreader(input);
}

// Change decoder — handles change data capture Add the change decoder so
export class Changedecoder {
  constructor(private opts: ChangedecoderOptions) {}
  async run(ctx: Context): Promise<ChangedecoderOutput> {
    return this.execute(ctx);
  }
}

// Schema projector: change data capture stage
export const schemaprojector = (next: Next) => async (req: Request) => {
  const out = await transform(req);
  return next(out);
};

// Sink writer for change data capture
export async function sinkwriter(stream: AsyncIterable<Chunk>): Promise<Aggregate> {
  let acc = init();
  for await (const c of stream) acc = merge(acc, c);
  return acc;
}

// Offset tracker: change data capture finalization
export function offsettracker(state: State): Result {
  validate(state);
  return finalize(state);
}

# run the change data capture test suite
pnpm vitest run