Global Sequential Testing for Multi-Stream Auditing

TL;DR

This paper introduces new sequential testing methods for multi-stream auditing that improve detection efficiency under various alternative hypotheses, balancing false alarm control and rapid detection.

Contribution

It develops merging test martingales and a balanced test that optimize expected stopping times for global hypothesis testing across multiple data streams.

Findings

Balanced test achieves $O(\frac{1}{k}\ln\frac{1}{\alpha})$ expected stopping time in dense settings.

Proposed tests outperform standard Bonferroni-based methods in simulations.

Effective on both synthetic and real-world datasets.

Abstract

Across many risk-sensitive areas, it is critical to continuously audit the performance of machine learning systems and detect any unusual behavior quickly. This can be modeled as a sequential hypothesis testing problem with $k$ incoming streams of data and a global null hypothesis that asserts that the system is working as expected across all $k$ streams. The standard global test employs a Bonferroni correction and has an expected stopping time bound of $O\left(\ln\frac{k}{\alpha}\right)$ when $k$ is large and the significance level of the test, $\alpha$, is small. In this work, we construct new sequential tests by using ideas of merging test martingales with different trade-offs in expected stopping times under different, sparse or dense alternative hypotheses. We further derive a new, balanced test that achieves an improved expected stopping time bound that matches Bonferroni's in the sparse setting but that naturally results in $O\left(\frac{1}{k}\ln\frac{1}{\alpha}\right)$ under a dense alternative. We empirically demonstrate the effectiveness of our proposed tests on synthetic and real-world data.