Bayesian vs Frequentist: Comparing Bayesian model selection with a frequentist approach using the iterative smoothing method

TL;DR

This paper compares a new frequentist model selection method with traditional Bayesian techniques using simulated supernova data, highlighting the frequentist approach's advantages in falsifying incorrect models.

Contribution

The paper introduces a frequentist model selection approach based on likelihood distributions from iterative smoothing, offering improved falsification capabilities over Bayesian methods.

Findings

Frequentist approach better at falsifying false models.

Bayesian approach can select the least incorrect model when all are false.

Simulated data demonstrates the effectiveness of the proposed method.

Abstract

We have developed a frequentist approach for model selection which determines the consistency between any cosmological model and the data using the distribution of likelihoods from the iterative smoothing method. Using this approach, we have shown how confidently we can conclude whether the data support any given model without comparison to a different one. In this current work, we compare our approach with the conventional Bayesian approach based on the estimation of the Bayesian evidence using nested sampling. We use simulated future Roman (formerly WFIRST)-like type Ia supernovae data in our analysis. We discuss the limits of the Bayesian approach for model selection and show how our proposed frequentist approach can perform better in the falsification of individual models. Namely, if the true model is among the candidates being tested in the Bayesian approach, that approach can select the correct model. If all of the options are false, then the Bayesian approach will select merely the least incorrect one. Our approach is designed for such a case and we can conclude that all of the models are false.