Sample Fit Reliability

TL;DR

This paper introduces Sample Fit Reliability (SFR), a novel set of methods to assess and improve the reliability of data sample fits by re-sampling and re-weighting observations, enhancing robustness and insight in empirical analyses.

Contribution

The paper develops SFR, a new computational framework that evaluates and enhances sample fit reliability through re-sampling, scoring, annealing, and re-weighting techniques, providing a complementary approach to existing model-focused methods.

Findings

SFR improves robustness of empirical results.

SFR reveals new insights in treatment effect studies.

Simulation demonstrates advantages of SFR.

Abstract

Researchers frequently test and improve model fit by holding a sample constant and varying the model. We propose methods to test and improve sample fit by holding a model constant and varying the sample. Much as the bootstrap is a well-known method to re-sample data and estimate the uncertainty of the fit of parameters in a model, we develop Sample Fit Reliability (SFR) as a set of computational methods to re-sample data and estimate the reliability of the fit of observations in a sample. SFR uses Scoring to assess the reliability of each observation in a sample, Annealing to check the sensitivity of results to removing unreliable data, and Fitting to re-weight observations for more robust analysis. We provide simulation evidence to demonstrate the advantages of using SFR, and we replicate three empirical studies with treatment effects to illustrate how SFR reveals new insights about each study.