Visual Inference and Graphical Representation in Regression Discontinuity Designs

TL;DR

This study investigates how effectively readers interpret regression discontinuity graphs, finding that graphical choices significantly influence perception and that visual inference can complement econometric methods.

Contribution

It provides empirical evidence on how graphical representation affects visual inference in RD designs and offers practical recommendations for constructing more effective RD graphs.

Findings

Small bins improve detection of discontinuities.

Fit lines can mislead perception of discontinuities.

Visual inference has comparable false positive rates to econometric methods.

Abstract

Despite the widespread use of graphs in empirical research, little is known about readers' ability to process the statistical information they are meant to convey ("visual inference"). We study visual inference within the context of regression discontinuity (RD) designs by measuring how accurately readers identify discontinuities in graphs produced from data generating processes calibrated on 11 published papers from leading economics journals. First, we assess the effects of different graphical representation methods on visual inference using randomized experiments. We find that bin widths and fit lines have the largest impacts on whether participants correctly perceive the presence or absence of a discontinuity. Our experimental results allow us to make evidence-based recommendations to practitioners, and we suggest using small bins with no fit lines as a starting point to construct RD graphs. Second, we compare visual inference on graphs constructed using our preferred method with widely used econometric inference procedures. We find that visual inference achieves similar or lower type I error (false positive) rates and complements econometric inference.