项目概述
本项目实现一个面向 alert evaluation 的完整系统,核心要解决的问题是 evaluating threshold and ratio rules over metric streams and firing alerts without flapping。整体由 Rule loader、Metric reader、Condition evaluator、Cooldown state、Notification dispatcher 五个模块组成,每个模块单一职责、可独立测试,最后通过集成测试验证端到端正确性。设计上把 a rule that flaps around its threshold and fires repeatedly because the cooldown is shorter than the metric jitter 作为第一类需求先行处理,而非事后再补。
架构与关键决策
数据模型为 a set of rules each with a condition over metric queries and a cooldown window that suppresses re-firing。Rule loader 定义其余阶段共同依赖的数据形状并在边界处校验输入;Metric reader 承载 alert evaluation 的正确性逻辑;Condition evaluator 专门处理 a rule that flaps around its threshold and fires repeatedly because the cooldown is shorter than the metric jitter,在失败路径上给出明确的拒绝、重试或降级策略;Cooldown state 提供可观测性,每次操作输出带关联 id 的结构化事件;Notification dispatcher 在过载时按既定方式降级,避免过载演变为数据损坏。配置与代码分离,使风险变更可经 diff 审阅。
实施与测试
按五个阶段依次实现 Rule loader、Metric reader、Condition evaluator、Cooldown state、Notification dispatcher,每阶段配套聚焦单测。集成测试组合各模块并驱动真实 alert evaluation 场景:先验证正常路径,再刻意触发 a rule that flaps around its threshold and fires repeatedly because the cooldown is shorter than the metric jitter 并断言系统降级而非损坏。所有依赖均在进程内或被桩替换,用例在 CI 中确定可复现。
限制与后续
- 单区域设计,多区域复制超出范围。
- Rule loader 默认进程内,未水平共享。
- 无鉴权层,假定部署在强制身份的边缘之后。
代码示例
# setup for alerting-rule-evaluator
pnpm init
pnpm add fastify undici redis
pnpm add -D typescript tsx vitest
alerting_rule_evaluator/
├── src/index.ts
├── src/rule-loader.ts
├── src/metric-reader.ts
├── tests/alerting_rule_evaluator.test.ts
└── package.json
// Rule loader: alert evaluation core
export function ruleloader(input: RuleloaderInput): RuleloaderResult {
// Build the rule loader that anchors alert evaluation and defines the core data shape.
return processRuleloader(input);
}
// Metric reader — handles alert evaluation Add the metric reader so
export class Metricreader {
constructor(private opts: MetricreaderOptions) {}
async run(ctx: Context): Promise<MetricreaderOutput> {
return this.execute(ctx);
}
}
// Condition evaluator: alert evaluation stage
export const conditionevaluator = (next: Next) => async (req: Request) => {
const out = await transform(req);
return next(out);
};
// Cooldown state for alert evaluation
export async function cooldownstate(stream: AsyncIterable<Chunk>): Promise<Aggregate> {
let acc = init();
for await (const c of stream) acc = merge(acc, c);
return acc;
}
// Notification dispatcher: alert evaluation finalization
export function notificationdispatcher(state: State): Result {
validate(state);
return finalize(state);
}
# run the alert evaluation test suite
pnpm vitest run