Anytime-valid sequential testing for elicitable functionals via supermartingales

TL;DR

This paper introduces a new method for sequential hypothesis testing of elicitable functionals using supermartingales, providing rigorous guarantees on test power and applicability to various data distributions.

Contribution

It develops a framework for anytime-valid sequential tests based on supermartingales for a broad class of nonparametric hypotheses involving elicitable functionals.

Findings

Provides asymptotic power guarantees for the tests.

Applicable to both bounded and unbounded data distributions.

Uses regret bounds from Online Convex Optimization.

Abstract

We design sequential tests for a large class of nonparametric null hypotheses based on elicitable and identifiable functionals. Such functionals are defined in terms of scoring functions and identification functions, which are ideal building blocks for constructing nonnegative supermartingales under the null. This in turn yields sequential tests via Ville's inequality. Using regret bounds from Online Convex Optimization, we obtain rigorous guarantees on the asymptotic power of the tests for a wide range of alternative hypotheses. Our results allow for bounded and unbounded data distributions, assuming that a sub-$\psi$ tail bound is satisfied.