Assumption Smuggling in Intermediate Outcome Tests of Causal Mechanisms

TL;DR

This paper demonstrates that intermediate outcome tests in causal mechanism analysis are fundamentally limited without strong assumptions, and proposes a method to evaluate the plausibility of these assumptions.

Contribution

It introduces a nonparametric bounding approach showing the limitations of intermediate outcome tests and offers a way to assess the key monotonicity assumption.

Findings

Intermediate outcome tests cannot definitively confirm or deny causal mechanisms without additional assumptions.

A new method to evaluate the plausibility of the monotonicity assumption is developed.

Bounds are estimated for recent experiments using intermediate outcome tests.

Abstract

Political scientists are increasingly interested in assessing causal mechanisms, or determining not just if a causal effect exists but also why it occurs. Even so, many researchers avoid formal causal mediation analyses due to their stringent assumptions, instead opting to explore causal mechanisms through what we call intermediate outcome tests. These tests estimate the effect of the treatment on one or more mediators and view such effects as suggestive evidence of a causal mechanism. In this paper, we use nonparametric bounding analysis to show that, without further assumptions, these tests can neither establish nor rule out the existence of a causal mechanism. To use intermediate outcome tests as a falsification test of causal mechanisms, researchers must make a very strong but rarely discussed monotonicity assumption. We develop a way to assess the plausibility of this monotonicity assumption and estimate our bounds for two recent experiments that use these tests.