Irreproducibility; Nothing is More Predictable

TL;DR

This paper demonstrates that subjective choices in data analysis significantly influence research outcomes, highlighting the challenges of irreproducibility and the importance of analysis methodology in observational studies.

Contribution

It shows how different analytical approaches yield varying inferences while maintaining similar predictive performance, emphasizing the role of subjective choices in scientific reproducibility.

Findings

Different analysis techniques lead to different inferences.

Models with different factors often have similar predictive accuracy.

Bayesian priors can help assess factor importance.

Abstract

The increasing ease of data capture and storage has led to a corresponding increase in the choice of data, the type of analysis performed on that data, and the complexity of the analysis performed. The main contribution of this paper is to show that the subjective choice of data and analysis methodology substantially impacts the identification of factors and outcomes of observational studies. This subjective variability of inference is at the heart of recent discussions around irreproducibility in scientific research. To demonstrate this subjective variability, data is taken from an existing study, where interest centres on understanding the factors associated with a young adult's propensity to fall into the category of `not in employment, education or training' (NEET). A fully probabilistic analysis is performed, set in a Bayesian framework and implemented using Reversible Jump Markov chain Monte Carlo (RJMCMC). The results show that different techniques lead to different inference but that models consisting of different factors often have the same predictive performance, whether the analysis is frequentist or Bayesian, making inference problematic. We demonstrate how the use of prior distributions in Bayesian techniques can be used to as a tool for assessing a factor's importance.