Comparing the frequentist and Bayesian periodic signal detection: rates of statistical mistakes and sensitivity to priors

TL;DR

This study compares frequentist and Bayesian methods for detecting periodic signals using Lomb-Scargle periodograms, finding similar detection efficiencies but highlighting the impact of prior choices and the conservative nature of Bayes factors.

Contribution

It provides a systematic comparison of Bayesian and frequentist approaches, emphasizing the effects of prior misspecification and the need for calibration of Bayes factors.

Findings

Bayesian and frequentist methods have nearly identical detection efficiency.

Nonuniform priors give a slight advantage, detecting about 1% more signals.

Bayes factors are often overconservative without proper calibration.

Abstract

We perform extensive Monte Carlo simulations to systematically compare the frequentist and Bayesian treatments of the Lomb--Scargle periodogram. The goal is to investigate whether the Bayesian period search is advantageous over the frequentist one in terms of the detection efficiency, how much if yes, and how sensitive it is regarding the choice of the priors, in particular in case of a misspecified prior (whenever the adopted prior does not match the actual distribution of physical objects). We find that the Bayesian and frequentist analyses always offer nearly identical detection efficiency in terms of their tradeoff between type-I and type-II mistakes. Bayesian detection may reveal a formal advantage if the frequency prior is nonuniform, but this results in only $\sim 1$ per cent extra detected signals. In case if the prior was misspecified (adopting nonuniform one over the actual uniform) this may turn into an opposite advantage of the frequentist analysis. Finally, we revealed that Bayes factor of this task appears rather overconservative if used without a calibration against type-I mistakes (false positives), thereby necessitating such a calibration in practice.