On the reliability of microvariability tests in quasars

TL;DR

This paper evaluates the reliability of statistical tests used to detect microvariability in quasars, highlighting the limitations of common methods and proposing improved approaches for more accurate results.

Contribution

It demonstrates the superiority of the Analysis of Variance test over traditional F-tests and suggests combining data from multiple stars to enhance detection reliability.

Findings

ANOVA test outperforms F-test in microvariability detection

Combining data from multiple comparison stars improves test reliability

Inappropriate use of certain tests can lead to unreliable detections

Abstract

Microvariations probe the physics and internal structure of quasars. Unpredictability and small flux variations make this phenomenon elusive and difficult to detect. Variance based probes such as the C and F tests, or a combination of both, are popular methods to compare the light-curves of the quasar and a comparison star. Recently, detection claims in some studies depend on the agreement of the results of the C and F tests, or of two instances of the F-test, in rejecting the non-variation null hypothesis. However, the C-test is a non-reliable statistical procedure, the F-test is not robust, and the combination of tests with concurrent results is anything but a straightforward methodology. A priori Power Analysis calculations and post hoc analysis of Monte-Carlo simulations show excellent agreement for the Analysis of Variance test to detect microvariations, as well as the limitations of the F-test. Additionally, combined tests yield correlated probabilities that make the assessment of statistical significance unworkable. However, it is possible to include data from several field stars to enhance the power in a single F-test, increasing the reliability of the statistical analysis. This would be the preferred methodology when several comparison stars are available. An example using two stars and the enhanced F-test is presented. These results show the importance of using adequate methodologies, and avoid inappropriate procedures that can jeopardize microvariability detections. Power analysis and Monte-Carlo simulations are useful tools for research planning, as they can reveal the robustness and reliability of different research approaches.