Sensitivity Checks in Pipeline Workflows

Sensitivity checks help you assess whether a causal estimate is robust or fragile. In quasi-experimental work, they are best treated as design diagnostics that probe assumptions and modelling choices, not as proofs that identification succeeded [Cook et al., 2002, Reichardt, 2019].

CausalPy’s pipeline API makes sensitivity analysis a first-class step, so robustness checks can run alongside model fitting and report generation in a single, reproducible workflow.

Architecture overview

The sensitivity framework has three main pieces:

1. Check — a protocol that individual checks implement. Each check declares which experiment types it applies to (applicable_methods), validates preconditions, and returns a structured CheckResult.

2. SensitivityAnalysis — a pipeline step that holds a list of Check objects and runs them against the fitted experiment.

3. CheckResult — the output of a check, containing a pass/fail verdict (or None when a check is informational or inconclusive), a prose summary, an optional diagnostics table, optional figures, and arbitrary metadata.

When a GenerateReport step follows SensitivityAnalysis, those results are included in the generated HTML report automatically.

Choosing checks

Start with the default suite

import causalpy as cp

result = cp.Pipeline(
    data=df,
    steps=[
        cp.EstimateEffect(
            method=cp.InterruptedTimeSeries,
            treatment_time=treatment_time,
            formula="y ~ 1 + t",
            model=cp.pymc_models.LinearRegression(),
        ),
        cp.SensitivityAnalysis.default_for(cp.InterruptedTimeSeries),
        cp.GenerateReport(),
    ],
).run()

SensitivityAnalysis.default_for(method) returns a pre-loaded step containing every check currently registered as a default for that experiment type. At present, PlaceboInTime is registered as the default check for InterruptedTimeSeries and SyntheticControl.

Compose a custom suite

cp.SensitivityAnalysis(
    checks=[
        cp.checks.PlaceboInTime(n_folds=4),
        cp.checks.PersistenceCheck(),
        cp.checks.PriorSensitivity(
            alternatives=[
                {"name": "diffuse", "model": cp.pymc_models.LinearRegression(...)},
            ]
        ),
    ]
)

SensitivityAnalysis checks applicability as it runs. If a check does not support the fitted experiment type, CausalPy raises a clear error naming the methods that check supports.

Quick reference

Check	Applies to	Registered as default?	Main question
PlaceboInTime	ITS, SC (PyMC models)	Yes, for ITS and SC	Do pseudo-interventions in the pre-period also produce “effects”?
PriorSensitivity	ITS, DiD, SC, Staggered DiD, RD, RKink, PrePostNEGD, IPW, IV (PyMC models)	No	Do conclusions change materially under reasonable prior alternatives?
PersistenceCheck	Three-period ITS designs	No	Does the effect remain after the intervention ends?
ConvexHullCheck	SC	No	Is the treated unit supported by the donor pool, or are we extrapolating?
LeaveOneOut	SC	No	Does the result depend heavily on one donor unit?
PlaceboInSpace	SC	No	Are placebo effects in control units as large as the treated effect?
BandwidthSensitivity	RD, RKink	No	Does the estimate depend heavily on bandwidth choice?
McCraryDensityTest	RD	No	Is there evidence of manipulation around the cutoff?
PreTreatmentPlaceboCheck	Staggered DiD	No	Do pre-treatment event-study effects look close to zero?

Where examples already exist

• PlaceboInTime: Pipeline Workflow, HTML Report Generation, Placebo-in-time and falsification: a UK CO₂ case study

• BandwidthSensitivity: Regression kink design with pymc models

• PreTreatmentPlaceboCheck: Staggered Difference-in-Differences

• More check-specific walkthroughs are still being added, so some checks currently have API coverage but no dedicated notebook example yet.

Check-by-check guide

PlaceboInTime

PlaceboInTime moves the intervention backward into the pre-treatment period and re-fits the model. If those pseudo-interventions often produce effects comparable to the observed one, the original result looks less credible. In synthetic control settings, placebo and falsification exercises are a standard part of design assessment [Abadie, 2021]; in interrupted time series settings, the same logic aligns with broader falsification practice in pre/post intervention designs [Lopez Bernal et al., 2017].

This check requires a PyMC-backed model because it works with posterior impact draws. Its identified placebo null calibrates the design and identification uncertainty that a posterior HDI does not contain; it does not replace the within-model practical/sign statement from HDI+ROPE. Use the two together: interpret HDI+ROPE within the fitted model, then ask whether the effect remains distinctive against the placebo null. The reporting statistics guide explains that division of responsibility.

For an identified null, cp.checks.operating_characteristics() computes the exact, deterministic conditional decision probabilities for each true effect size: correct detection, wrong-sign misclassification, and indeterminate outcomes under the configured ROPE rule. It is not an assurance calculation by itself. Assurance is the detection curve integrated against an explicit expected-effect prior, so it answers a design question only after you state which effects you expect. When PlaceboInTime is INCONCLUSIVE, no identified null exists and neither calibration-based operating characteristics nor assurance should be interpreted.

Fold eligibility and the pre-period constraint

Every placebo fold is a full re-fit, so a fold needs enough pre-treatment history to identify the model before its pseudo-intervention begins. PlaceboInTime requires each fold to have at least one full intervention window of observed history ahead of it, and skips folds that fall short rather than fitting them. A fold fitted on a handful of observations produces a posterior cumulative impact with a very large standard deviation, which inflates the between-fold scale tau of the hierarchical status-quo model and widens the learned null until it can swallow a genuine effect.

Skipped folds are reported in the check text and recorded in metadata["skipped_folds"] with the observed and required pre-period row counts, and a warning names the reason. If fewer than two usable folds remain, or the completed folds do not identify a finite positive between-fold spread, the check returns passed=None (inconclusive) rather than building an unidentified null. These results do not have metadata["null_samples"] or metadata["p_effect_outside_null"]; test those keys are present before accessing them, because a passed is None result has neither.

The constraint bites when the pre-period is only a few times longer than the post-period, because by default each fold consumes one post-period worth of history. There are two ways out:

• Pass an explicit intervention_length to shorten the placebo window. This shortens both the window and the history each fold requires, so more folds become eligible without changing the headline analysis. The actual effect is still summarised over the whole post-intervention period, so a much shorter window compares a long actual cumulative impact against a null built from short windows; PlaceboInTime warns when that happens and records both spans in metadata["comparison_window"].

• Pass an experiment_factory that adapts the model to the shorter fold data, for example by tightening priors or reducing the donor pool.

cp.checks.PlaceboInTime(n_folds=4, intervention_length=10, random_seed=42)

Reproducibility

PlaceboInTime has several stochastic stages: the per-fold experiment fits, the hierarchical status-quo pm.sample, the posterior predictive draw for theta_new, random fold selection, and the assurance simulation. When the check identifies a hierarchical null, the constructor’s random_seed is the master seed for all of them, so setting it alone is enough to make the reported verdict, metadata["p_effect_outside_null"] and metadata["null_samples"] reproducible. Folds are seeded as random_seed + fold_index, so they are independent of one another but stable across runs.

The one deliberate override is sample_kwargs["random_seed"]: when supplied explicitly it takes precedence, for the hierarchical pm.sample call only. Two cases sit outside the master seed and are surfaced rather than hidden. A custom experiment_factory owns any randomness it introduces, and an expected_effect_prior whose .rvs does not accept random_state is drawn unseeded, which raises a warning and is recorded in metadata["unseeded_custom_priors"].

PriorSensitivity

PriorSensitivity re-fits the same experiment with alternative prior specifications and compares the resulting effect summaries. Use it when prior choice could matter materially, especially in small samples or weakly identified models. Reporting how posterior conclusions change under reasonable alternatives is good Bayesian practice [Fan et al., 2023].

This is the broadest check in the current API, but it is only available for PyMC-backed experiments.

PersistenceCheck

PersistenceCheck applies to three-period ITS designs with treatment_end_time. It wraps analyze_persistence() to ask whether the effect persists, fades, or reverses after the intervention ends. This is especially relevant when policy or campaign effects may decay after treatment is removed [Wagner et al., 2002].

ConvexHullCheck

ConvexHullCheck asks whether the treated unit sits within the support of the donor pool in the pre-treatment period. If not, the synthetic control fit relies on extrapolation rather than interpolation, which weakens design credibility.

LeaveOneOut

LeaveOneOut drops one control unit at a time and re-fits the synthetic control. If the estimated effect changes dramatically when a single donor is removed, the result depends too heavily on that donor rather than on the donor pool as a whole.

PlaceboInSpace

PlaceboInSpace re-labels each control unit as though it were treated and compares those placebo effects to the observed treated effect. If many placebo units show effects as large as the treated unit, the original estimate looks less distinctive [Abadie et al., 2010].

BandwidthSensitivity

BandwidthSensitivity re-fits RD or RKink models across a sequence of bandwidths. Because bandwidth choice drives the bias-variance trade-off in local designs, a result that flips across plausible bandwidths should be treated cautiously [Lee and Lemieux, 2010, Imbens and Lemieux, 2008].

McCraryDensityTest

McCraryDensityTest checks for a discontinuity in the density of the running variable at the threshold. A sharp jump suggests units may have manipulated their assignment variable, undermining the design’s local comparability assumption [McCrary, 2008].

PreTreatmentPlaceboCheck

PreTreatmentPlaceboCheck examines pre-treatment event-study effects in staggered DiD. If negative event times are far from zero, the parallel trends story is harder to defend and the treatment effect may be biased [Goodman-Bacon, 2021, Borusyak et al., 2024].

Working with check results

Each check returns a CheckResult with the following fields:

• passed — True if the check passed, False if it failed, or None when it is informational or cannot reach a verdict.

• text — a prose summary describing the outcome.

• table — an optional pandas.DataFrame with diagnostic statistics.

• figures — an optional list of matplotlib figures.

• metadata — a dict of arbitrary extra data for downstream steps.

You can inspect results programmatically:

for cr in result.sensitivity_results:
    status = (
        "PASS"
        if cr.passed is True
        else ("FAIL" if cr.passed is False else "NO VERDICT")
    )
    print(f"[{status}] {cr.check_name}: {cr.text}")

    if cr.table is not None:
        display(cr.table)

When passed is None, inspect the result text: it may describe an informational diagnostic or an inconclusive check that could not reach a verdict.

Check-specific metadata can be absent when a check has no verdict. In particular, an inconclusive PlaceboInTime result has neither null_samples nor p_effect_outside_null; access learned-null values only after testing the relevant key, for example if "null_samples" in cr.metadata:.

When a GenerateReport step follows SensitivityAnalysis in the pipeline, check results are automatically included in the HTML report.

Interpreting sensitivity results

Important

Sensitivity checks are diagnostics, not definitive verdicts. A passing check does not prove your causal claim is correct, and a failing check does not prove it is wrong. They reveal where your analysis is robust and where it is fragile.

What a passing check tells you: The estimate survived a specific stress test. This increases confidence in the result, especially when multiple independent checks point in the same direction.

What a failing check tells you: The estimate is sensitive to a particular assumption or modelling choice. This does not invalidate the analysis; it tells you where to investigate further, justify your choices, or present stronger caveats.

General guidance:

• Start with the defaults, then add method-specific checks that target the most plausible failure modes for your design.

• Run more than one check. No single diagnostic is sufficient.

• Report failures as well as passes. Selective reporting of only passing checks undermines credibility.

• Use domain knowledge to decide which failures are consequential. A bandwidth warning at an extreme specification is different from strong placebo evidence.

• Treat checks as part of a cumulative argument, not a mechanical accept/reject gate.

Next steps

For the pipeline mechanics, see Pipeline Workflow. For HTML reporting of check results, see HTML Report Generation. More method-specific sensitivity walkthroughs will be added over time; where they already exist, they are linked above.

See also

• Pipeline Workflow — end-to-end pipeline tutorial

• HTML Report Generation — HTML report generation

• Staggered Difference-in-Differences — staggered DiD example with PreTreatmentPlaceboCheck

• Regression kink design with pymc models — regression kink example with BandwidthSensitivity

• Statistical Reporting in CausalPy — statistical concepts used in CausalPy reporting

References

[1]

Thomas D Cook, Donald Thomas Campbell, and William Shadish. Experimental and quasi-experimental designs for generalized causal inference. Volume 1195. Houghton Mifflin Boston, MA, 2002.

[2]

Charles S Reichardt. Quasi-experimentation: A guide to design and analysis. Guilford Publications, 2019.

[3]

Andrew Goodman-Bacon. Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2):254–277, 2021.

[4]

Alberto Abadie, Alexis Diamond, and Jens Hainmueller. Synthetic control methods for comparative case studies: estimating the effect of california's tobacco control program. Journal of the American Statistical Association, 105(490):493–505, 2010.

[5]

Alberto Abadie. Using synthetic controls: feasibility, data requirements, and methodological aspects. Journal of Economic Literature, 59(2):391–425, 2021.

[6]

Kirill Borusyak, Xavier Jaravel, and Jann Spiess. Revisiting event-study designs: robust and efficient estimation. Review of Economic Studies, 91(6):3253–3285, 2024.

[7]

Anita K Wagner, Stephen B Soumerai, Fang Zhang, and Dennis Ross-Degnan. Segmented regression analysis of interrupted time series studies in medication use research. Journal of Clinical Pharmacy and Therapeutics, 27(4):299–309, 2002.

[8]

James Lopez Bernal, Steven Cummins, and Antonio Gasparrini. Interrupted time series regression for the evaluation of public health interventions: a tutorial. International Journal of Epidemiology, 46(1):348–355, 2017.

[9]

David S Lee and Thomas Lemieux. Regression discontinuity designs in economics. Journal of Economic Literature, 48(2):281–355, 2010.

[10]

Justin McCrary. Manipulation of the running variable in the regression discontinuity design: a density test. Journal of Econometrics, 142(2):698–714, 2008.

[11]

Guido W Imbens and Thomas Lemieux. Regression discontinuity designs: a guide to practice. Journal of Econometrics, 142(2):615–635, 2008.

[12]

Li Fan, Ding Peng, and Mealli Fabrizia. Bayesian causal inference: a critical review. Philosophical Transactions of the Royal Society, 2023.