测试不稳定性度量

概念概述

测试不稳定性度量（Test Flakiness Metrics）指的是：不稳定性度量量化某测试在未修改代码上通过或失败的概率，把模糊感受转化为可排序的修复或隔离清单。

从更严格的定义看，Test flakiness metrics are measurements that estimate the probability a given test produces inconsistent results across otherwise identical runs, typically from repeated execution of the same revision. 该概念在工程实践中的价值，在于它能帮助团队用统一的语言描述一个具体的质量属性或设计约束，从而在评审与协作中减少歧义。

为什么需要它

They are used to prioritise which tests to fix or quarantine so that trust in the suite does not erode. 它与 Accessibility Tree Testing、The Test Oracle Problem、Branch Coverage 等概念经常一起出现，用于解决相近的工程问题。理解它的适用场景，有助于在多种备选方案之间做出有依据的取舍，而不是凭直觉选择技术方案。

常见误解

• a single retry pass proves a test is stable
• flaky tests can be ignored because they always eventually pass

需要避免的反模式

• quarantining flaky tests and never repairing them
• rerunning failed tests automatically without tracking the rate

实践要点

1. Repeat-running the same commit separates deterministic failures from nondeterministic ones, and the flip rate per test ranks candidates for investigation.
2. A flaky suite trains developers to ignore red builds, which hides genuine regressions, so flakiness is a reliability problem rather than a minor annoyance.
3. Fixing root causes such as timing assumptions, shared state, or uncontrolled clocks is durable; automatic retries merely mask the symptom.

参考来源

https://en.wikipedia.org/wiki/Flaky_test

代码示例

package calc

import "testing"

func Divide(a, b float64) (float64, bool) {
	if b == 0 {
		return 0, false
	}
	return a / b, true
}

func TestDivide(t *testing.T) {
	cases := []struct {
		name    string
		a, b    float64
		want    float64
		wantOK  bool
	}{
		{name: "even", a: 10, b: 2, want: 5, wantOK: true},
		{name: "fraction", a: 1, b: 4, want: 0.25, wantOK: true},
		{name: "divide by zero", a: 1, b: 0, want: 0, wantOK: false},
	}
	for _, tc := range cases {
		t.Run(tc.name, func(t *testing.T) {
			got, ok := Divide(tc.a, tc.b)
			if ok != tc.wantOK || got != tc.want {
				t.Fatalf("got (%v, %v), want (%v, %v)", got, ok, tc.want, tc.wantOK)
			}
		})
	}
}

package main

import (
	"errors"
	"testing"
)

var errNotFound = errors.New("order not found")

func ShippingCost(weightKg int, express bool, tier string) (int, error) {
	if weightKg <= 0 {
		return 0, errors.New("weight must be positive")
	}
	if weightKg > 30 {
		return 0, errors.New("weight exceeds courier limit")
	}

	base := 8
	if weightKg > 10 {
		base += (weightKg - 10) * 2
	}
	if express {
		base *= 2
	}
	if tier == "gold" {
		base -= 4
	}
	return base, nil
}

func TestShippingCost(t *testing.T) {
	t.Parallel()

	tests := []struct {
		name    string
		weight  int
		express bool
		tier    string
		want    int
		wantErr error
	}{
		{name: "light standard", weight: 2, express: false, tier: "", want: 8},
		{name: "light express", weight: 2, express: true, tier: "", want: 16},
		{name: "heavy adds per kg", weight: 15, express: false, tier: "", want: 18},
		{name: "heavy express gold", weight: 15, express: true, tier: "gold", want: 32},
		{name: "gold floor at zero", weight: 1, express: false, tier: "gold", want: 4},
		{name: "zero weight rejected", weight: 0, express: false, tier: "", wantErr: errors.New("weight must be positive")},
		{name: "over courier limit", weight: 31, express: false, tier: "", wantErr: errors.New("weight exceeds courier limit")},
	}

	for _, tt := range tests {
		tt := tt
		t.Run(tt.name, func(t *testing.T) {
			t.Parallel()

			got, err := ShippingCost(tt.weight, tt.express, tt.tier)
			if tt.wantErr != nil {
				if err == nil || err.Error() != tt.wantErr.Error() {
					t.Fatalf("ShippingCost(%d, %v, %q) error = %v, want %v",
						tt.weight, tt.express, tt.tier, err, tt.wantErr)
				}
				return
			}
			if err != nil {
				t.Fatalf("unexpected error: %v", err)
			}
			if got != tt.want {
				t.Errorf("ShippingCost(%d, %v, %q) = %d, want %d",
					tt.weight, tt.express, tt.tier, got, tt.want)
			}
		})
	}
}