Perturbed Datasets Methods for Hypothesis Testing and Structure of Corresponding Confidence Sets

TL;DR

This paper introduces a family of hypothesis testing methods that provide exact confidence levels without relying on specific noise distribution assumptions, including extensions of the sign-perturbed sums method.

Contribution

It generalizes the SPS method to a broader class of tests with exact confidence levels and analyzes the structure of resulting confidence sets in linear regression and dynamical systems.

Findings

Confidence regions can be connected, bounded, or non-convex.

SPS belongs to the new family of hypothesis tests.

Confidence sets for dynamical systems can be non-connected.

Abstract

Hypothesis testing methods that do not rely on exact distribution assumptions have been emerging lately. The method of sign-perturbed sums (SPS) is capable of characterizing confidence regions with exact confidence levels for linear regression and linear dynamical systems parameter estimation problems if the noise distribution is symmetric. This paper describes a general family of hypothesis testing methods that have an exact user chosen confidence level based on finite sample count and without relying on an assumed noise distribution. It is shown that the SPS method belongs to this family and we provide another hypothesis test for the case where the symmetry assumption is replaced with exchangeability. In the case of linear regression problems it is shown that the confidence regions are connected, bounded and possibly non-convex sets in both cases. To highlight the importance of understanding the structure of confidence regions corresponding to such hypothesis tests it is shown that confidence sets for linear dynamical systems parameter estimates generated using the SPS method can have non-connected parts, which have far reaching consequences.