Sequential Specification Tests to Choose a Model: A Change-Point Approach

TL;DR

This paper introduces a change-point approach using p-value distributions from sequential specification tests to robustly identify the optimal model specification without tuning parameters.

Contribution

It proposes a novel method based on change-point detection that is robust to errant p-values and does not require selecting a test level or tuning parameter.

Findings

Method accurately detects the point where assumptions break down.

Demonstrated effectiveness in bandwidth selection for regression discontinuity.

Shown to determine lag order in time series models.

Abstract

Researchers faced with a sequence of candidate model specifications must often choose the best specification that does not violate a testable identification assumption. One option in this scenario is sequential specification tests: hypothesis tests of the identification assumption over the sequence. Borrowing an idea from the change-point literature, this paper shows how to use the distribution of p-values from sequential specification tests to estimate the point in the sequence where the identification assumption ceases to hold. Unlike current approaches, this method is robust to individual errant p-values and does not require choosing a test level or tuning parameter. This paper demonstrates the method's properties with a simulation study, and illustrates it by application to the problems of choosing a bandwidth in a regression discontinuity design while maintaining covariate balance and of choosing a lag order for a time series model.