Calibrating the scan statistic: finite sample performance vs. asymptotics

TL;DR

This paper evaluates the finite sample performance of scan statistics for detecting signals in Gaussian and other distributions, proposing three calibration methods to improve practical detection across various signal lengths and settings.

Contribution

It introduces three novel calibration techniques for scan statistics that enhance finite sample performance and applicability across diverse distributional contexts.

Findings

Three calibration methods effectively improve detection power.

Calibration methods applicable to various distributions and null models.

Proposed methods are simple to implement and versatile.

Abstract

We consider the problem of detecting an elevated mean on an interval with unknown location and length in the univariate Gaussian sequence model. Recent results have shown that using scale-dependent critical values for the scan statistic allows to attain asymptotically optimal detection simultaneously for all signal lengths, thereby improving on the traditional scan, but this procedure has been criticized for losing too much power for short signals. We explain this discrepancy by showing that these asymptotic optimality results will necessarily be too imprecise to discern the performance of scan statistics in a practically relevant way, even in a large sample context. Instead, we propose to assess the performance with a new finite sample criterion. We then present three calibrations for scan statistics that perform well across a range of relevant signal lengths: The first calibration uses a particular adjustment to the critical values and is therefore tailored to the Gaussian case. The second calibration uses a scale-dependent adjustment to the significance levels and is therefore applicable to arbitrary known null distributions. The third calibration restricts the scan to a particular sparse subset of the scan windows and then applies a weighted Bonferroni adjustment to the corresponding test statistics. This {\sl Bonferroni scan} is also applicable to arbitrary null distributions and in addition is very simple to implement. We show how to apply these calibrations for scanning in a number of distributional settings: for normal observations with an unknown baseline and a known or unknown constant variance,for observations from a natural exponential family, for potentially heteroscadastic observations from a symmetric density by employing self-normalization in a novel way, and for exchangeable observations using tests based on permutations, ranks or signs.