When to Use Anytime-Valid Inference

Inference

AB Testing

A practical guide to anytime-valid inference in randomized experiments, early stopping, and decisions based on interim results.

Decisions based on interim A/B-test results

Suppose your team has finished a new feature and wants to test it with an A/B test. Before launch, you choose the primary KPI, the smallest effect worth detecting, and a 5% false-positive rate. The power calculation says that the experiment should run for twenty days, so the team fixes that runtime and launches.

The results are still visible every day. Whether the experimentation platform is built in-house or supplied by GrowthBook, Eppo, Statsig, or Optimizely, it usually updates the treatment estimate and p-value each morning.

On day one, the estimated effect is positive and the p-value is 0.15. On day two, the p-value is 0.08. On day three, it falls below 0.05 and remains there through day eight.

By day three, the team is already discussing whether to ship. Should we act on the result now, or wait until the planned twenty days are over?

Standard statistical inference does not support an unplanned early decision based on interim results. The 5% false-positive rate of an ordinary p-value is calibrated for one prespecified decision. If the team can make several decisions based on interim results, the chance that at least one of them crosses the threshold increases.

If twenty daily analyses were independent, the probability of at least one false positive would be \(1 - (1 - 0.05)^{20} \approx 0.64\). Daily analyses of accumulating data are not independent, so 64% is not the false-positive rate of the actual monitoring policy. In Scenario 1, we simulate these correlated daily analyses and measure the resulting false-positive rate. For the ordinary fixed-sample test, the 5% guarantee applies to the prespecified statistical test conducted at the end of the experiment.

The problem is not opening the dashboard every day. It arises when the interim results can trigger a decision to stop, ship, roll back, or extend the experiment.

Anytime-valid inference is designed for repeated monitoring with data-dependent decisions.

What SAVI changes

Safe anytime-valid inference (SAVI) supports decisions made while results accumulate. In the large randomized experiments considered here, a 5% SAVI test is calibrated so that, under the method’s assumptions, the probability that a null experiment ever crosses the decision boundary is asymptotically no more than 5%. The review schedule does not have to be chosen before launch. A prespecified rule can support an early ship or rollback decision without changing that error guarantee.

At the same sample size, a SAVI confidence sequence is generally wider than an ordinary fixed-sample confidence interval. When the true lift exceeds the effect used in the power calculation, SAVI may nevertheless reach a valid decision before day twenty. Effects near or below that target often require more data because the sequential boundary is stricter.

The five scenarios correspond to different product decisions. The first asks how often daily stopping creates a false positive; the second asks when a successful feature can ship early. The remaining scenarios use confidence sequences to approve a backend migration, report a long-term holdout every week, and decide whether to roll back a canary release.

PyFixest implements the linear-regression method of Lindon et al. (2026), which combines anytime-valid inference with regression adjustment for variance reduction.¹

import numpy as np
import pandas as pd
import plotly.graph_objects as go
from plotly.subplots import make_subplots

import pyfixest as pf

Show the simulation helpers

ALPHA = 0.05
N_DAYS = 20
USERS_PER_ARM_PER_DAY = 200
N_SIMULATIONS = 500
N_REPRESENTATIVE_PATHS = 100
PLANNED_MDE = 0.02
MAIN_TARGET_N = 2 * USERS_PER_ARM_PER_DAY * N_DAYS
MAIN_MIXTURE_PRECISION = pf.optimal_mixture_precision(
    nobs=MAIN_TARGET_N,
    number_of_coefficients=2,
    alpha=ALPHA,
)

SAVI_BLUE = "#2563EB"
ORDINARY_ORANGE = "#EA580C"
ESTIMATE_TEAL = "#0D9488"
NEUTRAL_GREY = "#64748B"


def _rgba(hex_color, opacity):
    r, g, b = (int(hex_color[i : i + 2], 16) for i in (1, 3, 5))
    return f"rgba({r}, {g}, {b}, {opacity})"


def _fit_at_day(data, with_intervals, mixture_precision):
    fit = pf.feols("converted ~ treated", data=data, vcov="hetero")
    row = {
        "estimate": float(fit.coef().loc["treated"]),
        "regular_pvalue": float(fit.pvalue().loc["treated"]),
        "savi_pvalue": float(
            fit.pvalue_savi(
                mixture_precision=mixture_precision,
            ).loc["treated"]
        ),
    }
    if with_intervals:
        ci = fit.confint(alpha=ALPHA).loc["treated"]
        cs = fit.confint(
            alpha=ALPHA,
            inference_type="savi",
            mixture_precision=mixture_precision,
        ).loc["treated"]
        row |= {
            "ci_lower": float(ci.iloc[0]),
            "ci_upper": float(ci.iloc[1]),
            "cs_lower": float(cs.iloc[0]),
            "cs_upper": float(cs.iloc[1]),
        }
    return row


def _experiment_path(
    p_control,
    p_treatment,
    seed,
    with_intervals=False,
    n_days=N_DAYS,
    users_per_day=USERS_PER_ARM_PER_DAY,
    mixture_precision=MAIN_MIXTURE_PRECISION,
):
    """One experiment: fit the model after each batch of users arrives."""
    rng = np.random.default_rng(seed)
    batches = []
    path = []
    for day in range(1, n_days + 1):
        treated = np.repeat([0, 1], users_per_day)
        converted = np.r_[
            rng.binomial(1, p_control, users_per_day),
            rng.binomial(1, p_treatment, users_per_day),
        ]
        order = rng.permutation(2 * users_per_day)
        batches.append(
            pd.DataFrame({"treated": treated[order], "converted": converted[order]})
        )
        row = {"day": day, "users": 2 * users_per_day * day}
        row |= _fit_at_day(
            pd.concat(batches, ignore_index=True),
            with_intervals,
            mixture_precision,
        )
        path.append(row)
    return pd.DataFrame(path)


def _simulate_paths(
    p_control,
    p_treatment,
    seed=42,
    n_simulations=N_SIMULATIONS,
    n_days=N_DAYS,
    mixture_precision=MAIN_MIXTURE_PRECISION,
):
    """Run many experiments; keep the full day-by-day estimate and p-value paths."""
    estimate = np.full((n_simulations, n_days), np.nan)
    regular_p = np.ones((n_simulations, n_days))
    savi_p = np.ones((n_simulations, n_days))
    for s in range(n_simulations):
        path = _experiment_path(
            p_control,
            p_treatment,
            seed + s,
            n_days=n_days,
            mixture_precision=mixture_precision,
        )
        estimate[s] = path["estimate"].to_numpy()
        regular_p[s] = path["regular_pvalue"].to_numpy()
        savi_p[s] = path["savi_pvalue"].to_numpy()
    return {"estimate": estimate, "regular_p": regular_p, "savi_p": savi_p}


def _stop_day(pmatrix):
    """First day (1-indexed) each row crosses ALPHA; 0 if it never does."""
    crossed = pmatrix <= ALPHA
    return np.where(crossed.any(axis=1), crossed.argmax(axis=1) + 1, 0)


def _representative_seed(stop_days, first_seed):
    """Choose a detected path whose stopping day is closest to the median."""
    detected = np.flatnonzero(stop_days > 0)
    median_day = np.median(stop_days[detected])
    index = detected[np.argmin(np.abs(stop_days[detected] - median_day))]
    return first_seed + int(index)


PLAY_PAUSE = [
    {
        "type": "buttons",
        "direction": "left",
        "x": 0,
        "y": -0.15,
        "xanchor": "left",
        "buttons": [
            {
                "label": "▶ Play",
                "method": "animate",
                "args": [
                    None,
                    {
                        "frame": {"duration": 500, "redraw": True},
                        "transition": {"duration": 150},
                        "fromcurrent": False,
                    },
                ],
            },
            {
                "label": "❚❚ Pause",
                "method": "animate",
                "args": [
                    [None],
                    {"frame": {"duration": 0, "redraw": False}, "mode": "immediate"},
                ],
            },
        ],
    }
]


def _slider(days, active):
    first_day, last_day = days[0], days[-1]
    return [
        {
            "active": active,
            "currentvalue": {"prefix": "Day "},
            "pad": {"t": 50},
            "steps": [
                {
                    "label": (
                        str(day)
                        if day in {first_day, last_day} or day % 5 == 0
                        else ""
                    ),
                    "method": "animate",
                    "args": [
                        [str(day)],
                        {
                            "frame": {"duration": 0, "redraw": True},
                            "mode": "immediate",
                        },
                    ],
                }
                for day in days
            ],
        }
    ]

Scenario 1: false positives from daily stopping

Before shipping on the day-three result from the introduction, we should ask how often the same rule would stop an experiment in which the treatment does nothing. We apply three decision rules to the same 500 simulated A/A experiments:

Use the ordinary p-value for the prespecified decision on day twenty.
Recompute the ordinary p-value every day and stop the first time it is at or below 0.05.
Apply the same daily stopping rule with the SAVI p-value.

Each experiment runs for twenty days, with 200 new users per arm each day. Conversion is 10% in both arms, so the true treatment effect is zero and every rejection is a false positive.

A fixed day-twenty test with \(\alpha = 0.05\) rejects a true null with probability about 5%. The figure shows the first 100 simulated experiments; the table below uses all 500. Each grey line is the cumulative treated-minus-control estimate, measured in percentage points, for one experiment. The estimates vary widely during the first few days and become more concentrated around zero as the sample grows.

The markers show what each policy would do on these same data. An orange cross marks the first day the ordinary daily p-value is at or below 0.05. A blue cross marks the first day the SAVI p-value is at or below 0.05. An orange diamond marks a rejection by the ordinary test on its prespecified analysis day, day twenty. A path can have more than one marker because all three policies are applied to every experiment.

Rejections occur when an estimate is large relative to its standard error. Stopping at the first ordinary p-value below 0.05 therefore selects large early deviations, including paths that later move back toward zero. The SAVI boundary accounts for the full sequence of daily decisions, which is why the figure contains far fewer blue crosses.

Show the simulation code

mirage = _simulate_paths(p_control=0.10, p_treatment=0.10, seed=42)

fpr_planned_analysis = (mirage["regular_p"][:, -1] <= ALPHA).mean()
fpr_daily_ordinary = (mirage["regular_p"] <= ALPHA).any(axis=1).mean()
fpr_daily_savi = (mirage["savi_p"] <= ALPHA).any(axis=1).mean()

Show the figure code

N_SHOWN = 100
EVIDENCE_BAR = -np.log10(ALPHA)
EVIDENCE_CAP = 4.0

show_est = mirage["estimate"][:N_SHOWN] * 100
reg_stop = _stop_day(mirage["regular_p"][:N_SHOWN])
savi_stop = _stop_day(mirage["savi_p"][:N_SHOWN])
final_reject = mirage["regular_p"][:N_SHOWN, -1] <= ALPHA

mirage_days = np.arange(1, N_DAYS + 1)
fig = go.Figure()
for experiment in range(N_SHOWN):
    fig.add_trace(
        go.Scatter(
            x=mirage_days,
            y=show_est[experiment],
            mode="lines",
            line={"color": _rgba(NEUTRAL_GREY, 0.3), "width": 1},
            name="One A/A path",
            showlegend=experiment == 0,
            hoverinfo="skip",
        )
    )

naive_fired = np.flatnonzero(reg_stop > 0)
savi_fired = np.flatnonzero(savi_stop > 0)
oneshot_fired = np.flatnonzero(final_reject)
fig.add_trace(
    go.Scatter(
        x=reg_stop[naive_fired],
        y=show_est[naive_fired, reg_stop[naive_fired] - 1],
        mode="markers",
        marker={"color": ORDINARY_ORANGE, "size": 9, "symbol": "x"},
        name=f"Naive daily stop ({naive_fired.size}/{N_SHOWN})",
        hovertemplate=(
            "Stopped on day %{x}<br>Reported effect: %{y:.1f} pp<extra></extra>"
        ),
    )
)
fig.add_trace(
    go.Scatter(
        x=savi_stop[savi_fired],
        y=show_est[savi_fired, savi_stop[savi_fired] - 1],
        mode="markers",
        marker={"color": SAVI_BLUE, "size": 11, "symbol": "x"},
        name=f"SAVI stop ({savi_fired.size}/{N_SHOWN})",
        hovertemplate=(
            "Stopped on day %{x}<br>Reported effect: %{y:.1f} pp<extra></extra>"
        ),
    )
)
fig.add_trace(
    go.Scatter(
        x=np.full(oneshot_fired.size, N_DAYS),
        y=show_est[oneshot_fired, -1],
        mode="markers",
        marker={
            "color": ORDINARY_ORANGE,
            "size": 10,
            "symbol": "diamond-open",
            "line": {"width": 2},
        },
        name=f"Planned day-20 rejection ({oneshot_fired.size}/{N_SHOWN})",
        hovertemplate="Rejected on day 20<br>Estimate: %{y:.1f} pp<extra></extra>",
    )
)
fig.add_hline(y=0, line_color="black", opacity=0.4)
fig.add_vline(
    x=N_DAYS,
    line_dash="dash",
    line_color=NEUTRAL_GREY,
    opacity=0.6,
    annotation_text="Planned end",
    annotation_position="top left",
    annotation_font_color=NEUTRAL_GREY,
)
fig.update_layout(
    title=(
        "100 A/A experiments, three decision rules"
        "<br>True effect is zero: every marked stop is a false positive"
    ),
    title_font_size=16,
    template="plotly_white",
    height=500,
    xaxis_title="Day of experiment",
    yaxis_title="Estimated effect (pp)",
    legend={"orientation": "h", "y": -0.22},
    margin={"l": 65, "r": 30, "t": 95, "b": 120},
)
fig.update_xaxes(dtick=2, range=[0.5, N_DAYS + 0.5])
fig

Effect paths for 100 simulated A/A experiments. Because the true effect is zero, every marked rejection is a false positive. Orange crosses show the first day the ordinary p-value falls to 0.05 or below under the daily stopping rule. Orange diamonds show rejections from the prespecified day-twenty analysis, and blue crosses show the first SAVI rejection.

Across all 500 experiments:

Show the table code

pd.DataFrame(
    {
        "Decision rule": [
            "Ordinary test, prespecified day-20 analysis",
            "Ordinary p-value, stop at first p ≤ 0.05",
            "SAVI p-value, stop at first p ≤ 0.05",
        ],
        "False-positive rate": [
            f"{fpr_planned_analysis:.1%}",
            f"{fpr_daily_ordinary:.1%}",
            f"{fpr_daily_savi:.1%}",
        ],
    }
)

	Decision rule	False-positive rate
0	Ordinary test, prespecified day-20 analysis	5.2%
1	Ordinary p-value, stop at first p ≤ 0.05	25.0%
2	SAVI p-value, stop at first p ≤ 0.05	1.0%

An ordinary analysis performed only on day twenty rejects 5.2% of the time, close to the intended 5%. Checking the ordinary p-value daily and stopping at the first rejection raises the rate to 25%. This is below the 64% back-of-the-envelope number from the introduction because the twenty analyses reuse the same accumulating data and are far from independent. It is nevertheless several times the specified 5%. SAVI rejects 1.0% of the same null experiments.

The SAVI false-positive rate need not equal 5%. Under the method’s assumptions, the probability of making a false rejection at any time is asymptotically at most 5%. The probability of rejecting by day twenty is therefore also at most 5% and can be lower.

The false-positive rate describes how often a stopping rule rejects when the true effect is zero. It does not describe the estimates reported when those rejections occur. Early estimates are noisy, and a rule that stops at the first p-value below 0.05 selects experiments in which noise produced an estimate that is large relative to its standard error.

For each experiment that rejects, the second plot records the estimate on its first rejection day. Experiments that never reject are omitted.

Show the simulation code

naive_stop = _stop_day(mirage["regular_p"])
savi_aa_stop = _stop_day(mirage["savi_p"])


def _estimate_at_stop(stop):
    idx = np.where(stop > 0)[0]
    return np.array([mirage["estimate"][i, stop[i] - 1] for i in idx]) * 100


naive_at_stop = _estimate_at_stop(naive_stop)
savi_at_stop = _estimate_at_stop(savi_aa_stop)

naive_rejection_rate = (naive_stop > 0).mean()
savi_rejection_rate = (savi_aa_stop > 0).mean()
naive_mean_abs = np.abs(naive_at_stop).mean()

Show the figure code

jitter = np.random.default_rng(1).uniform(-0.16, 0.16, size=N_SIMULATIONS)

fig = go.Figure()
for est, base, color, label in [
    (naive_at_stop, 1, ORDINARY_ORANGE, "Ordinary p-value stopping"),
    (savi_at_stop, 0, SAVI_BLUE, "SAVI"),
]:
    fig.add_trace(
        go.Scatter(
            x=est,
            y=base + jitter[: len(est)],
            mode="markers",
            marker={"color": _rgba(color, 0.5), "size": 7, "line": {"width": 0}},
            name=f"{label} ({len(est)}/{N_SIMULATIONS} rejected)",
            hovertemplate="Reported effect: %{x:.2f} pp<extra></extra>",
        )
    )
fig.add_vline(
    x=0,
    line_dash="dot",
    line_color=NEUTRAL_GREY,
    annotation_text="True effect: 0",
    annotation_position="top",
)
fig.update_layout(
    title="Estimates selected by each stopping rule<br>(true effect: 0)",
    template="plotly_white",
    height=380,
    xaxis_title="Estimated effect at stop (pp)",
    yaxis={
        "tickmode": "array",
        "tickvals": [0, 1],
        "ticktext": ["SAVI", "Ordinary p-value<br>stopping"],
        "range": [-0.5, 1.5],
    },
    legend={"orientation": "h", "y": -0.3},
    margin={"l": 80, "r": 30, "t": 90, "b": 105},
)
fig

Estimated effects at the first rejection, among experiments that reject within twenty days. Non-rejecting experiments are omitted.

The daily ordinary-p-value stopping rule rejects in 25% of these zero-effect experiments. Among the experiments that reject, the estimated effect at the stopping time averages 3.9 pp in absolute value even though the true effect is zero. These large magnitudes arise from selection at the stopping time.

SAVI requires stronger evidence because its decision boundary is calibrated for repeated monitoring. It rejects 1.0% of these null experiments, but the point estimate in an experiment that stops is still selected. A confidence sequence should therefore accompany the stopping decision.

Scenario 2: effect size and stopping time

Early stopping is most useful when the effect exceeds the value used in the power calculation. The A/B test was planned to have about 80% power to detect a lift from 10% to 12% after twenty days. In this simulation, conversion rises to 14% instead. The fixed-sample decision remains scheduled for day twenty, while SAVI can support an earlier decision once the data provide enough evidence.

We first follow one such experiment. Its SAVI stopping day is the median among the 100 simulated experiments that reject by day twenty.

The top panel tracks the estimated lift, an ordinary 95% confidence interval, and a 95% confidence sequence. The ordinary interval has pointwise coverage at a prespecified analysis time. The confidence sequence has time-uniform asymptotic coverage. The bottom panel shows the ordinary and SAVI p-values on a \(-\log_{10}(p)\) scale. This keeps both threshold crossings readable after the p-values become very small; the horizontal line corresponds to \(p=0.05\).

Show the simulation code

TRUE_LIFT = 2 * PLANNED_MDE
WINNER_FIRST_SEED = 20_000

winner_candidates = _simulate_paths(
    p_control=0.10,
    p_treatment=0.10 + TRUE_LIFT,
    seed=WINNER_FIRST_SEED,
    n_simulations=N_REPRESENTATIVE_PATHS,
)
winner_stop_days = _stop_day(winner_candidates["savi_p"])
winner_seed = _representative_seed(winner_stop_days, WINNER_FIRST_SEED)

winner = _experiment_path(
    p_control=0.10,
    p_treatment=0.10 + TRUE_LIFT,
    seed=winner_seed,
    with_intervals=True,
).set_index("day", drop=False)

peek_day = int(winner.loc[winner["regular_pvalue"] <= ALPHA, "day"].iloc[0])
savi_day = int(winner.loc[winner["savi_pvalue"] <= ALPHA, "day"].iloc[0])
stop_row = winner.loc[savi_day]
days_saved = N_DAYS - savi_day

peek_lift = f"{winner.loc[peek_day, 'estimate']:.1%}"
savi_users = f"{int(stop_row['users']):,}"
savi_lift = f"{stop_row['estimate']:.1%}"
savi_cs = f"[{stop_row['cs_lower']:.1%}, {stop_row['cs_upper']:.1%}]"

Show the animation code

winner_days = list(range(1, N_DAYS + 1))
shown = winner.loc[winner_days]
estimate_range = [
    float(shown["cs_lower"].min()) - 0.01,
    float(shown["cs_upper"].max()) + 0.01,
]


def _winner_traces(day):
    seen = winner.loc[winner["day"] <= day]
    current = winner.loc[day]
    band = {"mode": "lines", "line": {"width": 0}, "hoverinfo": "skip"}
    ordinary_crossed = day >= peek_day
    savi_crossed = day >= savi_day
    return [
        go.Scatter(x=seen["day"], y=seen["cs_upper"], showlegend=False, **band),
        go.Scatter(
            x=seen["day"],
            y=seen["cs_lower"],
            fill="tonexty",
            fillcolor=_rgba(SAVI_BLUE, 0.15),
            name="95% confidence sequence",
            **band,
        ),
        go.Scatter(x=seen["day"], y=seen["ci_upper"], showlegend=False, **band),
        go.Scatter(
            x=seen["day"],
            y=seen["ci_lower"],
            fill="tonexty",
            fillcolor=_rgba(ORDINARY_ORANGE, 0.18),
            name="95% confidence interval",
            **band,
        ),
        go.Scatter(
            x=[day, day],
            y=[current["cs_lower"], current["cs_upper"]],
            mode="lines+markers",
            line={"color": SAVI_BLUE, "width": 3},
            marker={"color": SAVI_BLUE, "size": 5},
            showlegend=False,
            hovertemplate=(
                f"Day {day}<br>Confidence sequence: "
                f"[{current['cs_lower']:.1%}, {current['cs_upper']:.1%}]"
                "<extra></extra>"
            ),
        ),
        go.Scatter(
            x=[day, day],
            y=[current["ci_lower"], current["ci_upper"]],
            mode="lines+markers",
            line={"color": ORDINARY_ORANGE, "width": 2},
            marker={"color": ORDINARY_ORANGE, "size": 4},
            showlegend=False,
            hovertemplate=(
                f"Day {day}<br>Ordinary interval: "
                f"[{current['ci_lower']:.1%}, {current['ci_upper']:.1%}]"
                "<extra></extra>"
            ),
        ),
        go.Scatter(
            x=seen["day"],
            y=seen["estimate"],
            mode="lines+markers",
            line_color=ESTIMATE_TEAL,
            name="Estimated lift",
            hovertemplate="Day %{x}<br>Lift: %{y:.1%}<extra></extra>",
        ),
        go.Scatter(
            x=seen["day"],
            y=np.minimum(-np.log10(seen["regular_pvalue"]), EVIDENCE_CAP),
            mode="lines+markers",
            line={"color": ORDINARY_ORANGE, "dash": "dot"},
            name="Ordinary p-value",
            customdata=seen["regular_pvalue"],
            hovertemplate="Day %{x}<br>p = %{customdata:.4f}<extra></extra>",
        ),
        go.Scatter(
            x=seen["day"],
            y=np.minimum(-np.log10(seen["savi_pvalue"]), EVIDENCE_CAP),
            mode="lines+markers",
            line_color=SAVI_BLUE,
            name="SAVI p-value",
            customdata=seen["savi_pvalue"],
            hovertemplate="Day %{x}<br>p = %{customdata:.4f}<extra></extra>",
        ),
        go.Scatter(
            x=[peek_day] if ordinary_crossed else [None],
            y=[
                min(
                    -np.log10(winner.loc[peek_day, "regular_pvalue"]),
                    EVIDENCE_CAP,
                )
            ]
            if ordinary_crossed
            else [None],
            mode="markers",
            marker={"color": ORDINARY_ORANGE, "size": 10, "symbol": "diamond"},
            showlegend=False,
            hoverinfo="skip",
        ),
        go.Scatter(
            x=[savi_day] if savi_crossed else [None],
            y=[
                min(
                    -np.log10(winner.loc[savi_day, "savi_pvalue"]),
                    EVIDENCE_CAP,
                )
            ]
            if savi_crossed
            else [None],
            mode="markers",
            marker={"color": SAVI_BLUE, "size": 10, "symbol": "diamond"},
            showlegend=False,
            hoverinfo="skip",
        ),
        go.Scatter(
            x=[peek_day, peek_day] if ordinary_crossed else [None, None],
            y=[0, EVIDENCE_CAP],
            mode="lines",
            line={"color": ORDINARY_ORANGE, "dash": "dot", "width": 1.5},
            showlegend=False,
            hovertemplate=(
                f"Day {peek_day}: ordinary p-value below 0.05;"
                " fixed-sample stop invalid<extra></extra>"
            ),
        ),
        go.Scatter(
            x=[savi_day, savi_day] if savi_crossed else [None, None],
            y=[0, EVIDENCE_CAP],
            mode="lines",
            line={"color": SAVI_BLUE, "dash": "dot", "width": 1.5},
            showlegend=False,
            hovertemplate=(
                f"Day {savi_day}: SAVI p-value below 0.05;"
                " stopping is valid<extra></extra>"
            ),
        ),
    ]


def _winner_title(day):
    users = int(winner.loc[day, "users"])
    if day < peek_day:
        decision = "Continue"
    elif day < savi_day:
        decision = "Ordinary p < 0.05; no fixed-sample decision yet"
    else:
        decision = "SAVI boundary reached; stopping is valid"
    return f"Day {day} of {N_DAYS} · {users:,} users<br>{decision}"


fig = make_subplots(
    rows=2,
    cols=1,
    shared_xaxes=True,
    vertical_spacing=0.12,
    subplot_titles=("Estimated lift", "Evidence against 'no effect'"),
)
for i, trace in enumerate(_winner_traces(winner_days[0])):
    fig.add_trace(trace, row=1 if i < 7 else 2, col=1)
fig.frames = [
    go.Frame(
        data=_winner_traces(day),
        traces=list(range(13)),
        name=str(day),
        layout={"title": {"text": _winner_title(day)}},
    )
    for day in winner_days
]
fig.add_hline(
    y=TRUE_LIFT,
    line_dash="dot",
    line_color=NEUTRAL_GREY,
    annotation_text=f"True lift: {TRUE_LIFT:.0%}",
    row=1,
    col=1,
)
fig.add_hline(
    y=EVIDENCE_BAR,
    line_dash="dash",
    line_color="black",
    annotation_text="Decision bar (p = 0.05)",
    row=2,
    col=1,
)
fig.add_vline(
    x=N_DAYS,
    line_dash="dash",
    line_color=NEUTRAL_GREY,
    opacity=0.7,
    annotation_text="Planned end: first valid fixed-sample decision",
    annotation_position="top left",
    annotation_font_color=NEUTRAL_GREY,
    row=2,
    col=1,
)
fig.update_layout(
    title=_winner_title(winner_days[0]),
    title_font_size=16,
    template="plotly_white",
    height=740,
    xaxis_range=[0.5, N_DAYS + 0.5],
    xaxis2_range=[0.5, N_DAYS + 0.5],
    yaxis_range=estimate_range,
    yaxis2_range=[0, EVIDENCE_CAP],
    yaxis_tickformat=".0%",
    yaxis2_title="−log10(p-value), capped at 4",
    xaxis2_title="Day of experiment",
    legend={"orientation": "h", "y": -0.48},
    updatemenus=PLAY_PAUSE,
    sliders=_slider(winner_days, active=0),
    margin={"l": 70, "r": 35, "t": 125, "b": 270},
)
fig.update_xaxes(dtick=2)
fig

One simulated experiment with a true lift of 4 pp. The orange and blue vertical lines mark the first ordinary and SAVI p-values at or below 0.05. The evidence axis is capped at 4 so that the decision boundary remains visible.

The ordinary p-value falls below 0.05 on day 3, when the estimated lift is 4.5%. The ordinary procedure was calibrated for the prespecified day-twenty decision. Replacing that decision with a rule that stops at the first p-value below 0.05 gives sampling variation twenty opportunities to cross the threshold, so the orange line does not retain the planned 5% false-positive guarantee.

The SAVI p-value falls below 0.05 on day 7, after 2,800 users. Its confidence sequence is [0.3%, 8.0%]. The prespecified SAVI rule is met with 13 days left in the planned runtime.

The confidence sequence should be reported with the stopping decision. A rule that stops when evidence is strong selects estimates that look favorable, so the point estimate on the stopping day tends to overstate the true lift. The confidence sequence retains its coverage at that stopping time.

A second simulation follows true lifts of 1, 2, and 4 percentage points for forty days to show how the effect size changes the stopping time:

Show the simulation code

MDE_MONITORING_DAYS = 40

handcuffs = _simulate_paths(
    p_control=0.10,
    p_treatment=0.10 + PLANNED_MDE,
    seed=10_000,
    n_days=MDE_MONITORING_DAYS,
)
smaller_effect_paths = _simulate_paths(
    p_control=0.10,
    p_treatment=0.11,
    seed=11_000,
    n_days=MDE_MONITORING_DAYS,
)
larger_effect_paths = _simulate_paths(
    p_control=0.10,
    p_treatment=0.14,
    seed=12_000,
    n_days=MDE_MONITORING_DAYS,
)

savi_detected = np.maximum.accumulate(
    (handcuffs["savi_p"] <= ALPHA) & (handcuffs["estimate"] > 0),
    axis=1,
)
savi_detection = savi_detected.mean(axis=0)
smaller_savi_detection = np.maximum.accumulate(
    (smaller_effect_paths["savi_p"] <= ALPHA)
    & (smaller_effect_paths["estimate"] > 0),
    axis=1,
).mean(axis=0)
larger_savi_detection = np.maximum.accumulate(
    (larger_effect_paths["savi_p"] <= ALPHA)
    & (larger_effect_paths["estimate"] > 0),
    axis=1,
).mean(axis=0)
oneshot_power = (
    (handcuffs["regular_p"][:, N_DAYS - 1] <= ALPHA)
    & (handcuffs["estimate"][:, N_DAYS - 1] > 0)
).mean()

Show the figure code

days = np.arange(1, MDE_MONITORING_DAYS + 1)
oneshot_curve = np.where(days >= N_DAYS, oneshot_power, 0.0)

fig = go.Figure()
fig.add_trace(
    go.Scatter(
        x=days,
        y=oneshot_curve,
        mode="lines+markers",
        line={"shape": "hv", "dash": "dash", "color": ORDINARY_ORANGE},
        name="Fixed test: 2 pp, day 20",
        hovertemplate="Day %{x}<br>Detected: %{y:.0%}<extra></extra>",
    )
)
for detection, dash, marker, label in [
    (smaller_savi_detection, "dot", "circle-open", "SAVI: 1 pp lift"),
    (savi_detection, "dash", "diamond-open", "SAVI: 2 pp lift (MDE)"),
    (larger_savi_detection, "solid", "circle", "SAVI: 4 pp lift"),
]:
    fig.add_trace(
        go.Scatter(
            x=days,
            y=detection,
            mode="lines+markers",
            line={"color": SAVI_BLUE, "dash": dash},
            marker={"symbol": marker},
            name=label,
            hovertemplate=(
                "Day %{x}<br>Valid positive decision by this day: "
                "%{y:.0%}<extra></extra>"
            ),
        )
    )
fig.add_vline(
    x=N_DAYS,
    line_dash="dash",
    line_color=NEUTRAL_GREY,
    opacity=0.7,
    annotation_text="Planned end: day 20",
    annotation_position="bottom right",
    annotation_font_color=NEUTRAL_GREY,
)
fig.update_layout(
    title="SAVI stopping time by true lift",
    xaxis_title="Day of experiment",
    yaxis_title="Share with a valid positive decision",
    yaxis_tickformat=".0%",
    yaxis_range=[0, 1],
    hovermode="x unified",
    template="plotly_white",
    legend={"orientation": "h", "y": -0.3},
    height=500,
    margin={"l": 75, "r": 30, "t": 90, "b": 145},
)
fig.update_xaxes(dtick=5, range=[1, MDE_MONITORING_DAYS])
fig

Share of SAVI experiments with a valid positive decision by each day, for true lifts of 1, 2, and 4 pp. The fixed-test curve reports power for a true 2 pp lift at its prespecified decision time, day twenty.

At day twenty, the fixed-horizon test detects the effect in 80% of experiments when the true lift is the planned 2 pp. By then, SAVI has stopped in 99% of experiments with a 4 pp lift, 49% with a 2 pp lift, and 8% with a 1 pp lift. Because SAVI remains valid after the planned endpoint, the experiment can continue. By day forty, those shares are 100%, 86%, and 23%, respectively.

Many 4 pp effects cross the SAVI boundary before day twenty. Effects at the 2 pp MDE frequently need longer, and 1 pp effects take longer still. At the MDE, SAVI has lower day-twenty power than the fixed-horizon test because its boundary remains valid when the experiment continues and the stopping time depends on the data.

Scenario 3: Backend Migration

A backend migration poses a different decision. A checkout service is moving to a new backend, and the team will approve it only after the data rule out a conversion change of 1 pp or more in either direction. Approval requires an interval contained within -1 to +1 pp.

The team reviews the result each day and certifies the migration as soon as the confidence sequence lies inside that acceptance range. The certification day is determined by the results, so an ordinary fixed-time confidence interval cannot support this rule. The A/A-style test sends 2,000 users per arm per day to the old and new backends. The displayed path has the median confidence-sequence certification day among 100 simulations that certify within thirty days.

Show the animation code

DELTA = 0.01
BACKEND_USERS = 2_000
BACKEND_MAX_DAYS = 30
BACKEND_FIRST_SEED = 50_000
BACKEND_MIXTURE_PRECISION = pf.optimal_mixture_precision(
    nobs=2 * BACKEND_USERS * BACKEND_MAX_DAYS,
    number_of_coefficients=2,
    alpha=ALPHA,
)

backend_cert_days = np.zeros(N_REPRESENTATIVE_PATHS, dtype=int)
backend_ci_cert_days = np.zeros(N_REPRESENTATIVE_PATHS, dtype=int)
for simulation in range(N_REPRESENTATIVE_PATHS):
    candidate = _experiment_path(
        p_control=0.10,
        p_treatment=0.10,
        seed=BACKEND_FIRST_SEED + simulation,
        with_intervals=True,
        n_days=BACKEND_MAX_DAYS,
        users_per_day=BACKEND_USERS,
        mixture_precision=BACKEND_MIXTURE_PRECISION,
    )
    candidate_certified = (
        (candidate["cs_lower"] > -DELTA)
        & (candidate["cs_upper"] < DELTA)
    )
    if candidate_certified.any():
        backend_cert_days[simulation] = int(
            candidate.loc[candidate_certified, "day"].iloc[0]
        )
    candidate_ci_certified = (
        (candidate["ci_lower"] > -DELTA)
        & (candidate["ci_upper"] < DELTA)
    )
    if candidate_ci_certified.any():
        backend_ci_cert_days[simulation] = int(
            candidate.loc[candidate_ci_certified, "day"].iloc[0]
        )

median_cs_cert = int(np.median(backend_cert_days[backend_cert_days > 0]))
median_ci_cert = int(np.median(backend_ci_cert_days[backend_ci_cert_days > 0]))

backend_seed = _representative_seed(
    backend_cert_days,
    BACKEND_FIRST_SEED,
)

backend_full = _experiment_path(
    p_control=0.10,
    p_treatment=0.10,
    seed=backend_seed,
    with_intervals=True,
    n_days=BACKEND_MAX_DAYS,
    users_per_day=BACKEND_USERS,
    mixture_precision=BACKEND_MIXTURE_PRECISION,
).set_index("day", drop=False)

certified = (backend_full["cs_lower"] > -DELTA) & (backend_full["cs_upper"] < DELTA)
cert_day = int(backend_full.loc[certified, "day"].iloc[0])
cert_cs = (
    f"[{backend_full.loc[cert_day, 'cs_lower']:.2%}, "
    f"{backend_full.loc[cert_day, 'cs_upper']:.2%}]"
)
ci_certified = (backend_full["ci_lower"] > -DELTA) & (backend_full["ci_upper"] < DELTA)
ci_cert_day = int(backend_full.loc[ci_certified, "day"].iloc[0])

backend_path = backend_full.loc[backend_full["day"] <= cert_day]
b_range = [
    float(backend_path["cs_lower"].min()) - 0.005,
    float(backend_path["cs_upper"].max()) + 0.005,
]


def _backend_traces(day):
    seen = backend_path.loc[backend_path["day"] <= day]
    current = backend_path.loc[day]
    band = {"mode": "lines", "line": {"width": 0}, "hoverinfo": "skip"}
    tolerance_met = day >= cert_day
    return [
        go.Scatter(x=seen["day"], y=seen["cs_upper"], showlegend=False, **band),
        go.Scatter(
            x=seen["day"],
            y=seen["cs_lower"],
            fill="tonexty",
            fillcolor=_rgba(SAVI_BLUE, 0.15),
            name="95% confidence sequence",
            **band,
        ),
        go.Scatter(x=seen["day"], y=seen["ci_upper"], showlegend=False, **band),
        go.Scatter(
            x=seen["day"],
            y=seen["ci_lower"],
            fill="tonexty",
            fillcolor=_rgba(ORDINARY_ORANGE, 0.18),
            name="95% confidence interval",
            **band,
        ),
        go.Scatter(
            x=[day, day],
            y=[current["cs_lower"], current["cs_upper"]],
            mode="lines+markers",
            line={"color": SAVI_BLUE, "width": 3},
            marker={"color": SAVI_BLUE, "size": 5},
            showlegend=False,
            hovertemplate=(
                f"Day {day}<br>Confidence sequence: "
                f"[{current['cs_lower']:.2%}, {current['cs_upper']:.2%}]"
                "<extra></extra>"
            ),
        ),
        go.Scatter(
            x=[day, day],
            y=[current["ci_lower"], current["ci_upper"]],
            mode="lines+markers",
            line={"color": ORDINARY_ORANGE, "width": 2},
            marker={"color": ORDINARY_ORANGE, "size": 4},
            showlegend=False,
            hovertemplate=(
                f"Day {day}<br>Ordinary interval: "
                f"[{current['ci_lower']:.2%}, {current['ci_upper']:.2%}]"
                "<extra></extra>"
            ),
        ),
        go.Scatter(
            x=seen["day"],
            y=seen["estimate"],
            mode="lines+markers",
            line_color=ESTIMATE_TEAL,
            name="Estimated effect",
            hovertemplate="Day %{x}<br>Effect: %{y:.1%}<extra></extra>",
        ),
        go.Scatter(
            x=[cert_day] if tolerance_met else [None],
            y=[backend_path.loc[cert_day, "estimate"]] if tolerance_met else [None],
            mode="markers+text",
            marker={"color": SAVI_BLUE, "size": 11, "symbol": "diamond"},
            text=["Tolerance met"] if tolerance_met else [None],
            textposition="bottom left",
            showlegend=False,
            hoverinfo="skip",
        ),
        go.Scatter(
            x=[ci_cert_day, ci_cert_day] if day >= ci_cert_day else [None, None],
            y=b_range,
            mode="lines",
            line={"color": ORDINARY_ORANGE, "dash": "dot", "width": 2},
            showlegend=False,
            hovertemplate=(
                f"Day {ci_cert_day}: ordinary interval inside tolerance;"
                " certification invalid<extra></extra>"
            ),
        ),
        go.Scatter(
            x=[cert_day, cert_day] if tolerance_met else [None, None],
            y=b_range,
            mode="lines",
            line={"color": SAVI_BLUE, "dash": "dot", "width": 2},
            showlegend=False,
            hovertemplate=(
                f"Day {cert_day}: confidence sequence inside tolerance;"
                " certification valid<extra></extra>"
            ),
        ),
    ]


def _backend_title(day):
    users = int(backend_path.loc[day, "users"])
    if day == cert_day:
        decision = "CS inside ±1 pp: tolerance met, test over"
    elif day >= ci_cert_day:
        decision = "Ordinary interval inside tolerance; continue"
    else:
        decision = "Tolerance not met; keep collecting"
    return f"Day {day} · {users:,} users<br>{decision}"


backend_days = list(range(1, cert_day + 1))
fig = go.Figure(_backend_traces(1))
fig.frames = [
    go.Frame(
        data=_backend_traces(day),
        name=str(day),
        layout={"title": {"text": _backend_title(day)}},
    )
    for day in backend_days
]
fig.add_hrect(
    y0=-DELTA,
    y1=DELTA,
    fillcolor=_rgba(SAVI_BLUE, 0.07),
    line_width=0,
)
fig.add_hline(
    y=DELTA,
    line_dash="dash",
    line_color=NEUTRAL_GREY,
    opacity=0.7,
    annotation_text="Certification tube: ±1 pp",
    annotation_position="top right",
    annotation_font_color=NEUTRAL_GREY,
)
fig.add_hline(y=-DELTA, line_dash="dash", line_color=NEUTRAL_GREY, opacity=0.7)
fig.add_hline(y=0, line_color="black", opacity=0.4)
fig.update_layout(
    title=_backend_title(1),
    title_font_size=16,
    template="plotly_white",
    height=600,
    xaxis_title="Day of A/A test",
    xaxis_range=[0.5, cert_day + 0.5],
    yaxis_title="Estimated effect",
    yaxis_range=b_range,
    yaxis_tickformat=".0%",
    legend={"orientation": "h", "y": -0.48},
    updatemenus=PLAY_PAUSE,
    sliders=_slider(backend_days, active=0),
    margin={"l": 70, "r": 35, "t": 105, "b": 245},
)
fig.update_xaxes(dtick=2)
fig

The blue vertical line marks the first day the confidence sequence lies entirely inside the -1 to +1 pp acceptance range. The orange line marks the first day the ordinary confidence interval does the same.

On day 13, the confidence sequence [-0.96%, 0.68%] lies inside the ±1 pp tolerance. Its time-uniform asymptotic coverage allows the team to stop and rule out effects larger than the tolerance.

In this run, the ordinary interval enters the tolerance on day 5, earlier than the confidence sequence on day 13. An ordinary interval provides pointwise 95% coverage when the analysis time is fixed independently of the results. Here, however, the team selects the first day on which the interval fits inside the acceptance range. Repeated checks make it more likely that sampling variation produces a favorable interval on at least one day. The interval selected on that day is no longer a fixed-time 95% interval.

Among harmless migrations, the median certification day is 7 for the ordinary rule and 13 for the confidence sequence. The earlier ordinary dates reflect narrower pointwise intervals, not valid early certification.

The harmless A/A design measures certification time, but not certification error: approving a zero-effect migration is correct. To estimate false certification, we repeat the simulation at -1 pp, the nearest effect defined as unacceptable. Every certification in this boundary design is an error; effects farther outside the range are easier to distinguish.

Show the simulation code

BOUNDARY_FIRST_SEED = 70_000

boundary_ci_certified = np.zeros(N_REPRESENTATIVE_PATHS, dtype=bool)
boundary_cs_certified = np.zeros(N_REPRESENTATIVE_PATHS, dtype=bool)
for simulation in range(N_REPRESENTATIVE_PATHS):
    candidate = _experiment_path(
        p_control=0.10,
        p_treatment=0.10 - DELTA,
        seed=BOUNDARY_FIRST_SEED + simulation,
        with_intervals=True,
        n_days=BACKEND_MAX_DAYS,
        users_per_day=BACKEND_USERS,
        mixture_precision=BACKEND_MIXTURE_PRECISION,
    )
    boundary_ci_certified[simulation] = bool(
        ((candidate["ci_lower"] > -DELTA) & (candidate["ci_upper"] < DELTA)).any()
    )
    boundary_cs_certified[simulation] = bool(
        ((candidate["cs_lower"] > -DELTA) & (candidate["cs_upper"] < DELTA)).any()
    )

false_cert_ci = boundary_ci_certified.mean()
false_cert_cs = boundary_cs_certified.mean()

Show the table code

pd.DataFrame(
    {
        "Stopping rule": [
            "Ordinary interval: certify at first entry",
            "Confidence sequence: certify at first entry",
        ],
        "False certification rate": [
            f"{false_cert_ci:.0%}",
            f"{false_cert_cs:.0%}",
        ],
    }
)

	Stopping rule	False certification rate
0	Ordinary interval: certify at first entry	12%
1	Confidence sequence: certify at first entry	1%

The ordinary-interval rule falsely certifies 12% of the boundary-effect migrations. The confidence-sequence rule falsely certifies 1%. If the confidence sequence contains the true -1 pp effect on every day, it cannot lie strictly inside the -1 to +1 pp acceptance range. A false certification can occur only when the confidence sequence misses the true effect, which has asymptotic probability at most 5% under the method’s assumptions.

With only 100 simulated migrations, each false certification changes the estimated rate by one percentage point, so the reported rates are only rough estimates.

Failing to reject a zero effect would not certify the migration either. A non-significant p-value can still be consistent with effects outside the acceptance range. Certification requires the full confidence sequence to lie inside that range.

Scenario 4: monitor a long-term holdout

A product area may keep a small randomized group of users from receiving new features for an entire year. Comparing this holdout with exposed users measures the cumulative effect of everything the area has shipped. The estimate is reported at each weekly business review. There is no stopping rule here; the interval itself must remain valid across repeated reports.

Each ordinary confidence interval has 95% pointwise coverage at a fixed review time, but the collection of weekly intervals does not have 95% simultaneous coverage. As the number of reviews grows, so does the probability that at least one interval misses the true effect. A 95% confidence sequence is constructed to contain the true effect at every review with asymptotic probability of at least 95%, under the method’s assumptions and regardless of the monitoring schedule.

The holdout simulation assumes that the shipped features lift conversion from 10% to 11%. Each week adds 500 users per arm, and the team reviews the result weekly for a year.

Show the simulation code

HOLDOUT_WEEKS = 52
HOLDOUT_USERS_PER_WEEK = 500
HOLDOUT_LIFT = 0.01
HOLDOUT_SIMULATIONS = 100
HOLDOUT_FIRST_SEED = 60_000
HOLDOUT_MIXTURE_PRECISION = pf.optimal_mixture_precision(
    nobs=2 * HOLDOUT_USERS_PER_WEEK * HOLDOUT_WEEKS,
    number_of_coefficients=2,
    alpha=ALPHA,
)

holdout_bounds = {
    key: np.empty((HOLDOUT_SIMULATIONS, HOLDOUT_WEEKS))
    for key in ("estimate", "ci_lower", "ci_upper", "cs_lower", "cs_upper")
}
for simulation in range(HOLDOUT_SIMULATIONS):
    path = _experiment_path(
        p_control=0.10,
        p_treatment=0.10 + HOLDOUT_LIFT,
        seed=HOLDOUT_FIRST_SEED + simulation,
        with_intervals=True,
        n_days=HOLDOUT_WEEKS,
        users_per_day=HOLDOUT_USERS_PER_WEEK,
        mixture_precision=HOLDOUT_MIXTURE_PRECISION,
    )
    for key in holdout_bounds:
        holdout_bounds[key][simulation] = path[key].to_numpy()

ci_covers = (holdout_bounds["ci_lower"] <= HOLDOUT_LIFT) & (
    holdout_bounds["ci_upper"] >= HOLDOUT_LIFT
)
cs_covers = (holdout_bounds["cs_lower"] <= HOLDOUT_LIFT) & (
    holdout_bounds["cs_upper"] >= HOLDOUT_LIFT
)
ci_always_covered = np.logical_and.accumulate(ci_covers, axis=1).mean(axis=0)
cs_always_covered = np.logical_and.accumulate(cs_covers, axis=1).mean(axis=0)
ci_ever_missed = 1 - ci_always_covered
cs_ever_missed = 1 - cs_always_covered

holdout_weeks_axis = np.arange(1, HOLDOUT_WEEKS + 1)
holdout_index = int(
    np.argsort(holdout_bounds["estimate"][:, -1])[HOLDOUT_SIMULATIONS // 2]
)
holdout_path = pd.DataFrame(
    {key: values[holdout_index] for key, values in holdout_bounds.items()}
).assign(week=holdout_weeks_axis)
holdout_final = holdout_path.iloc[-1]
holdout_cs = f"[{holdout_final['cs_lower']:.2%}, {holdout_final['cs_upper']:.2%}]"
holdout_ci = f"[{holdout_final['ci_lower']:.2%}, {holdout_final['ci_upper']:.2%}]"

Show the figure code

band = {"mode": "lines", "line": {"width": 0}, "hoverinfo": "skip"}
fig = go.Figure()
fig.add_trace(
    go.Scatter(
        x=holdout_path["week"],
        y=holdout_path["cs_upper"],
        showlegend=False,
        **band,
    )
)
fig.add_trace(
    go.Scatter(
        x=holdout_path["week"],
        y=holdout_path["cs_lower"],
        fill="tonexty",
        fillcolor=_rgba(SAVI_BLUE, 0.15),
        name="95% confidence sequence",
        **band,
    )
)
fig.add_trace(
    go.Scatter(
        x=holdout_path["week"],
        y=holdout_path["ci_upper"],
        showlegend=False,
        **band,
    )
)
fig.add_trace(
    go.Scatter(
        x=holdout_path["week"],
        y=holdout_path["ci_lower"],
        fill="tonexty",
        fillcolor=_rgba(ORDINARY_ORANGE, 0.18),
        name="95% confidence interval",
        **band,
    )
)
fig.add_trace(
    go.Scatter(
        x=holdout_path["week"],
        y=holdout_path["estimate"],
        mode="lines+markers",
        line_color=ESTIMATE_TEAL,
        name="Estimated lift",
        hovertemplate="Week %{x}<br>Lift: %{y:.2%}<extra></extra>",
    )
)
fig.add_hline(y=0, line_color="black", opacity=0.4)
fig.add_hline(
    y=HOLDOUT_LIFT,
    line_dash="dot",
    line_color=NEUTRAL_GREY,
    annotation_text="True lift: 1 pp",
    annotation_position="top right",
)
fig.add_annotation(
    x=3,
    y=0.029,
    text="Early intervals extend beyond this range",
    showarrow=False,
    xanchor="left",
    font={"color": NEUTRAL_GREY},
)
fig.update_layout(
    title="A long-term holdout, reviewed weekly for one year",
    template="plotly_white",
    height=430,
    xaxis_title="Week of holdout",
    yaxis_title="Estimated lift",
    yaxis_tickformat=".1%",
    yaxis_range=[-0.02, 0.032],
    legend={"orientation": "h", "y": -0.3},
    margin={"l": 70, "r": 35, "t": 70, "b": 120},
)
fig

One simulated holdout, reviewed weekly for a year; the y-axis clips the very wide intervals of the first weeks. The confidence sequence is constructed for simultaneous coverage across all reviews, while the ordinary interval is calibrated for one prespecified review.

After a year, this holdout reports a confidence sequence of [0.18%, 1.82%] and an ordinary interval of [0.47%, 1.53%]. One path shows how the intervals evolve, but not how often either method misses the true lift. Across the 100 simulated holdouts, we record whether each interval has contained the true lift at every review so far:

Show the figure code

fig = go.Figure()
fig.add_trace(
    go.Scatter(
        x=holdout_weeks_axis,
        y=ci_ever_missed,
        mode="lines+markers",
        line={"color": ORDINARY_ORANGE, "dash": "dot"},
        name="Ordinary 95% confidence interval",
        hovertemplate="Week %{x}<br>Missed at least once: %{y:.0%}<extra></extra>",
    )
)
fig.add_trace(
    go.Scatter(
        x=holdout_weeks_axis,
        y=cs_ever_missed,
        mode="lines+markers",
        line_color=SAVI_BLUE,
        name="95% confidence sequence",
        hovertemplate="Week %{x}<br>Missed at least once: %{y:.0%}<extra></extra>",
    )
)
fig.add_hline(
    y=ALPHA,
    line_dash="dash",
    line_color="black",
    annotation_text="5% simultaneous miss-rate boundary",
    annotation_position="top right",
)
fig.update_layout(
    title="Cumulative interval miss rate across weekly reviews",
    template="plotly_white",
    height=430,
    xaxis_title="Week of holdout",
    yaxis_title="Missed at least once",
    yaxis_tickformat=".0%",
    yaxis_range=[0, 0.35],
    hovermode="x unified",
    legend={"orientation": "h", "y": -0.3},
    margin={"l": 75, "r": 35, "t": 70, "b": 120},
)
fig

Share of simulated holdouts whose interval has missed the true lift at least once by each weekly review.

By week 52, the ordinary interval has missed the true lift at least once in 29% of the simulated holdouts. The confidence sequence did not miss in any of these 100 runs. That does not imply a zero miss probability: 100 runs are too few to estimate a 5% rate precisely. The cumulative miss rate for the ordinary interval can only stay flat or rise as the reviews continue.

Scenario 5: decide whether to roll back a canary

A canary release sends a small share of traffic to a new service version before a full rollout. Waiting for a fixed two-day analysis can leave a harmful version in production. Acting on the first ordinary interval that falls entirely below the rollback boundary would make the analysis time depend on the data. A confidence sequence supports the same decision after each hourly batch.

The team will tolerate a conversion loss of no more than 1 pp, so -1 pp is the decision boundary. With 500 new users per arm each hour:

Roll back if the upper endpoint is below -1 pp; every effect compatible with the data implies a loss greater than 1 pp.
Promote if the lower endpoint is above -1 pp; a loss of 1 pp or more has been ruled out.
Continue while the sequence contains -1 pp.

The simulated version reduces conversion from 10% to 7%, a true effect of -3 pp. The correct decision is to roll back; the figure shows how many hourly batches are needed before the data support it.

Show the simulation code

CANARY_USERS_PER_HOUR = 500
CANARY_MAX_HOURS = 48
CANARY_HARM_TOLERANCE = 0.01
CANARY_MIXTURE_PRECISION = pf.optimal_mixture_precision(
    nobs=2 * CANARY_USERS_PER_HOUR * CANARY_MAX_HOURS,
    number_of_coefficients=2,
    alpha=ALPHA,
)

canary_full = _experiment_path(
    p_control=0.10,
    p_treatment=0.07,
    seed=8,
    with_intervals=True,
    n_days=CANARY_MAX_HOURS,
    users_per_day=CANARY_USERS_PER_HOUR,
    mixture_precision=CANARY_MIXTURE_PRECISION,
).set_index("day", drop=False)

rollback_hour = int(
    canary_full.loc[
        canary_full["cs_upper"] < -CANARY_HARM_TOLERANCE,
        "day",
    ].iloc[0]
)
rollback_cs = (
    f"[{canary_full.loc[rollback_hour, 'cs_lower']:.2%}, "
    f"{canary_full.loc[rollback_hour, 'cs_upper']:.2%}]"
)
hours_saved = CANARY_MAX_HOURS - rollback_hour
canary_path = canary_full.loc[canary_full["day"] <= rollback_hour]

Show the figure code

band = {"mode": "lines", "line": {"width": 0}, "hoverinfo": "skip"}
fig = go.Figure()
fig.add_trace(
    go.Scatter(x=canary_path["day"], y=canary_path["cs_upper"], showlegend=False, **band)
)
fig.add_trace(
    go.Scatter(
        x=canary_path["day"],
        y=canary_path["cs_lower"],
        fill="tonexty",
        fillcolor=_rgba(SAVI_BLUE, 0.15),
        name="95% confidence sequence",
        **band,
    )
)
fig.add_trace(
    go.Scatter(
        x=canary_path["day"],
        y=canary_path["estimate"],
        mode="lines+markers",
        line_color=ESTIMATE_TEAL,
        name="Estimated effect",
        hovertemplate="Hour %{x}<br>Effect: %{y:.1%}<extra></extra>",
    )
)
fig.add_hline(y=0, line_color="black", opacity=0.4)
fig.add_hline(
    y=-CANARY_HARM_TOLERANCE,
    line_dash="dash",
    line_color=ORDINARY_ORANGE,
    annotation_text="Decision boundary: −1 pp",
    annotation_position="top left",
)
fig.add_hline(
    y=-0.03,
    line_dash="dot",
    line_color=NEUTRAL_GREY,
    annotation_text="True effect: −3 pp",
    annotation_position="bottom left",
)
fig.add_vline(
    x=rollback_hour,
    line_dash="dash",
    line_color=ORDINARY_ORANGE,
    opacity=0.7,
    annotation_text=f"Hour {rollback_hour}: roll back",
    annotation_position="top right",
    annotation_font_color=ORDINARY_ORANGE,
)
fig.add_vline(
    x=CANARY_MAX_HOURS,
    line_dash="dash",
    line_color=NEUTRAL_GREY,
    opacity=0.7,
    annotation_text="Planned 2-day runtime",
    annotation_position="bottom left",
    annotation_font_color=NEUTRAL_GREY,
)
fig.update_layout(
    title="A harmful canary release",
    template="plotly_white",
    height=430,
    xaxis_title="Hour since release",
    xaxis_range=[0.5, CANARY_MAX_HOURS + 0.5],
    yaxis_title="Estimated effect on conversion",
    yaxis_tickformat=".0%",
    yaxis_range=[-0.085, 0.025],
    legend={"orientation": "h", "y": -0.3},
    margin={"l": 70, "r": 35, "t": 70, "b": 120},
)
fig

A canary with a true effect of -3 pp. Rollback occurs when the confidence sequence’s upper endpoint falls below the -1 pp boundary.

The sequence starts wide because one hour of data provides little information. After 15 hours it lies entirely below the -1 pp boundary, at [-3.88%, -1.03%], and the rule calls for a rollback. A fixed-horizon test with a planned two-day runtime would wait another 33 hours before making its planned decision.

Regression adjustment can narrow the sequence

The examples so far use only a treatment indicator. In practice, a pre-treatment metric that predicts the outcome can reduce residual variation, narrow the confidence sequence, and help the test reach its stopping rule sooner. The method of Lindon et al. makes this possible because it applies anytime-valid inference to regression coefficients.

We compare adjusted and unadjusted analyses over twenty days in 200 experiments. The true average treatment effect is 0.18. The data-generating process is nonlinear, heteroskedastic, and heavy-tailed; the treatment effect also varies with the pre-experiment metric. Neither fitted regression matches the true outcome model.

The adjusted regression includes the pre-experiment metric and its interaction with treatment. Because the metric is centered, the coefficient on treated targets the average treatment effect. Both regressions use robust standard errors, and neither regression includes the quadratic term or the changing residual scale used to generate the data. For each day, we compare the median confidence-sequence half-width and the share of experiments that meet the stopping rule. This teaching simulation isolates the precision gain from regression adjustment under model misspecification.

Show the regression-adjustment simulation

ADJUSTMENT_SIMULATIONS = 200
ADJUSTMENT_NULL_SIMULATIONS = 200
ADJUSTMENT_USERS_PER_DAY = 200
ADJUSTMENT_EFFECT = 0.18
ADJUSTMENT_TARGET_N = N_DAYS * ADJUSTMENT_USERS_PER_DAY
ADJUSTMENT_MIXTURE_PRECISION = pf.optimal_mixture_precision(
    nobs=ADJUSTMENT_TARGET_N,
    number_of_coefficients=4,
    alpha=ALPHA,
)


def _adjustment_path(
    seed,
    average_effect=ADJUSTMENT_EFFECT,
    labels=("unadjusted", "adjusted"),
):
    rng = np.random.default_rng(seed)
    batches = []
    path = []
    for day in range(1, N_DAYS + 1):
        pre_metric = rng.normal(size=ADJUSTMENT_USERS_PER_DAY)
        treated = rng.binomial(1, 0.5, size=ADJUSTMENT_USERS_PER_DAY)
        residual = rng.standard_t(df=3, size=ADJUSTMENT_USERS_PER_DAY) / np.sqrt(3)
        outcome = (
            1.5 * pre_metric
            + 0.5 * (pre_metric**2 - 1)
            + treated * (average_effect + 0.12 * pre_metric)
            + (0.8 + 0.25 * pre_metric**2) * residual
        )
        batches.append(
            pd.DataFrame(
                {
                    "outcome": outcome,
                    "treated": treated,
                    "pre_metric": pre_metric,
                }
            )
        )
        data = pd.concat(batches, ignore_index=True)
        row = {"day": day}
        for label, formula in [
            ("unadjusted", "outcome ~ treated"),
            ("adjusted", "outcome ~ treated * pre_metric"),
        ]:
            if label not in labels:
                continue
            fit = pf.feols(formula, data=data, vcov="hetero")
            cs = fit.confint(
                alpha=ALPHA,
                inference_type="savi",
                mixture_precision=ADJUSTMENT_MIXTURE_PRECISION,
            ).loc["treated"]
            row[f"{label}_half_width"] = float((cs.iloc[1] - cs.iloc[0]) / 2)
            row[f"{label}_pvalue"] = float(
                fit.pvalue_savi(
                    mixture_precision=ADJUSTMENT_MIXTURE_PRECISION
                ).loc["treated"]
            )
            row[f"{label}_estimate"] = float(fit.coef().loc["treated"])
        path.append(row)
    return pd.DataFrame(path)


adjustment_width = {
    "unadjusted": np.empty((ADJUSTMENT_SIMULATIONS, N_DAYS)),
    "adjusted": np.empty((ADJUSTMENT_SIMULATIONS, N_DAYS)),
}
adjustment_pvalue = {
    "unadjusted": np.empty((ADJUSTMENT_SIMULATIONS, N_DAYS)),
    "adjusted": np.empty((ADJUSTMENT_SIMULATIONS, N_DAYS)),
}
adjustment_estimate = {
    "unadjusted": np.empty((ADJUSTMENT_SIMULATIONS, N_DAYS)),
    "adjusted": np.empty((ADJUSTMENT_SIMULATIONS, N_DAYS)),
}

for simulation in range(ADJUSTMENT_SIMULATIONS):
    path = _adjustment_path(seed=30_000 + simulation)
    for label in ("unadjusted", "adjusted"):
        adjustment_width[label][simulation] = path[
            f"{label}_half_width"
        ].to_numpy()
        adjustment_pvalue[label][simulation] = path[f"{label}_pvalue"].to_numpy()
        adjustment_estimate[label][simulation] = path[
            f"{label}_estimate"
        ].to_numpy()

adjustment_null_pvalue = np.empty((ADJUSTMENT_NULL_SIMULATIONS, N_DAYS))
for simulation in range(ADJUSTMENT_NULL_SIMULATIONS):
    path = _adjustment_path(
        seed=40_000 + simulation,
        average_effect=0,
        labels=("adjusted",),
    )
    adjustment_null_pvalue[simulation] = path["adjusted_pvalue"].to_numpy()

adjustment_days = np.arange(1, N_DAYS + 1)
median_width = {
    label: np.median(adjustment_width[label], axis=0)
    for label in ("unadjusted", "adjusted")
}
stopped_share = {
    label: np.maximum.accumulate(
        adjustment_pvalue[label] <= ALPHA,
        axis=1,
    ).mean(axis=0)
    for label in ("unadjusted", "adjusted")
}
mean_estimate = {
    label: adjustment_estimate[label][:, -1].mean()
    for label in ("unadjusted", "adjusted")
}
adjustment_null_fire_rate = (
    adjustment_null_pvalue <= ALPHA
).any(axis=1).mean()
adjustment_null_rejections = (
    adjustment_null_pvalue <= ALPHA
).any(axis=1).sum()

Show the figure code

fig = make_subplots(
    rows=2,
    cols=1,
    shared_xaxes=True,
    vertical_spacing=0.15,
    subplot_titles=(
        "Median 95% confidence-sequence half-width",
        "Share of experiments that can stop",
    ),
)

for label, color, dash, display in [
    ("unadjusted", NEUTRAL_GREY, "dot", "Without adjustment"),
    ("adjusted", SAVI_BLUE, "solid", "With fully interacted adjustment"),
]:
    fig.add_trace(
        go.Scatter(
            x=adjustment_days,
            y=median_width[label],
            mode="lines+markers",
            line={"color": color, "dash": dash},
            name=display,
            hovertemplate="Day %{x}<br>Half-width: %{y:.3f}<extra></extra>",
        ),
        row=1,
        col=1,
    )
    fig.add_trace(
        go.Scatter(
            x=adjustment_days,
            y=stopped_share[label],
            mode="lines+markers",
            line={"color": color, "dash": dash},
            name=display,
            showlegend=False,
            hovertemplate="Day %{x}<br>Stopped: %{y:.0%}<extra></extra>",
        ),
        row=2,
        col=1,
    )

fig.add_hline(
    y=ADJUSTMENT_EFFECT,
    line_dash="dash",
    line_color=NEUTRAL_GREY,
    annotation_text="Reference: half-width = ATE magnitude (0.18)",
    annotation_position="bottom right",
    row=1,
    col=1,
)
fig.update_layout(
    template="plotly_white",
    height=590,
    hovermode="x unified",
    legend={"orientation": "h", "y": -0.18},
    margin={"l": 75, "r": 35, "t": 75, "b": 105},
)
fig.update_yaxes(title_text="Half-width", row=1, col=1)
fig.update_yaxes(
    title_text="Stopped",
    tickformat=".0%",
    range=[0, 1],
    row=2,
    col=1,
)
fig.update_xaxes(title_text="Day of experiment", dtick=2, row=2, col=1)
fig

The same experiments analyzed with and without a pre-treatment covariate. A predictive covariate narrows the confidence sequence (top) and raises the share of experiments that can stop by the planned end (bottom). Under the method’s assumptions, both analyses have the same coefficient-wise asymptotic guarantee.

By day twenty, the median confidence-sequence half-width falls from 0.195 without adjustment to 0.124 with adjustment. The adjusted analysis meets the stopping rule in 96% of experiments, compared with 50% without the pre-experiment metric.

Both treatment estimates remain centered on the randomized treatment effect despite the misspecified outcome models. At day twenty, the mean treatment estimates are 0.183 without adjustment and 0.185 with adjustment, against a true average effect of 0.180. The covariate improves precision by predicting the outcome.

To check whether adjustment inflates false positives in this design, we repeat the simulation with an average treatment effect of zero. Across 200 experiments, the SAVI p-value fell below 0.05 by day twenty in 2 runs (1.0%). This check would reveal a large size distortion, but 200 runs cannot estimate a 5% rate precisely.

Regression adjustment should use only variables measured before treatment. Conditioning on a post-treatment variable can bias the treatment coefficient.

Apply SAVI in PyFixest

The scenarios use different decision rules, but the implementation is the same. Before monitoring, choose the outcome, regression specification, covariance estimator, and tested coefficient, and keep them fixed. At each review, append the newly completed experimental units and refit the same regression. Here, treated is the randomized treatment indicator and converted is the primary metric.

rng = np.random.default_rng(42)
n_per_arm = 800

experiment_data = pd.DataFrame(
    {
        "treated": np.repeat([0, 1], n_per_arm),
        "converted": np.r_[
            rng.binomial(1, 0.10, n_per_arm),
            rng.binomial(1, 0.18, n_per_arm),
        ],
    }
)

fit = pf.feols("converted ~ treated", data=experiment_data, vcov="hetero")

mixture_precision = pf.optimal_mixture_precision(
    nobs=8_000,
    number_of_coefficients=2,
    alpha=0.05,
)

At a 5% threshold, the decision rule for treated has three equivalent forms:

# Rule 1: stop when the sequential p-value is at most 0.05.
fit.pvalue_savi(
    mixture_precision=mixture_precision,
).loc["treated"]

np.float64(1.3060511149198845e-05)

# Rule 2: stop when the e-value is at least 20 (= 1 / 0.05).
fit.evalue(
    mixture_precision=mixture_precision,
).loc["treated"]

np.float64(76566.6816999993)

# Rule 3: stop when zero falls outside the confidence sequence.
fit.confint(
    inference_type="savi",
    mixture_precision=mixture_precision,
).loc["treated"]

2.5%     0.049847
97.5%    0.167653
Name: treated, dtype: float64

Since the sequential p-value is min(1, 1 / e_value), the three rules always agree for a given coefficient. A dashboard can show whichever unit its audience prefers without changing the decision. Confidence sequences are often the easiest to act on because they put the result and any practical tolerance on the same scale.

The mixture_precision argument controls where the sequence is most sensitive over time. Here it is chosen to minimize sequence width at the planned 8,000 users. The default is 1, but whichever value you use must be chosen before monitoring and held fixed.

What the guarantee assumes

The examples in this guide assume that treatment is randomized and that the rows entering the analysis are independent experimental units, such as users. Randomization identifies the causal effect. SAVI supplies time-uniform inference for the chosen coefficient; it provides no causal identification for an observational comparison.

When treatment is assigned by user, the analysis should use independent user-level observations. Repeated page views from the same user are not independent users. Define one fixed outcome window per user and append the user-level outcome only after that window is complete. PyFixest’s SAVI accessors do not currently support clustered repeated observations.

PyFixest uses the asymptotic t-mixture version of SAVI. It is intended for large experiments where the usual regression standard-error approximation is credible, not as a finite-sample guarantee. The test is coefficient-wise: the three accessors above refer to one regression coefficient, not a joint F test.

What SAVI does not fix

SAVI controls repeated data-dependent decisions for one chosen test. It does not correct for testing many outcomes, subgroups, variants, or coefficients. If your experiment has several primary comparisons, pair SAVI with a multiple-testing procedure.

The implementation follows Lindon, Ham, Tingley, and Bojinov (2026), “Anytime-Valid Inference in Linear Models with Applications to Regression-Adjusted Causal Inference”. The paper gives the derivation and a detailed account of the assumptions.

Footnotes

Its closed-form formulas also make the method fast to compute.↩︎