A Unified Framework for Testing High Dimensional Parameters: A Data-Adaptive Approach

TL;DR

This paper introduces a unified, data-adaptive framework for high-dimensional hypothesis testing that combines multiple tests to be effective across various alternative scenarios, utilizing $U$-statistics and bootstrap methods.

Contribution

It develops a broad family of tests based on a novel $L_p$-norm variant and constructs a data-adaptive test that is powerful under diverse high-dimensional alternatives.

Findings

Proposed tests are asymptotically optimal.

The framework effectively handles both dense and sparse alternatives.

Numerical results demonstrate superior performance on simulated and real data.

Abstract

High dimensional hypothesis test deals with models in which the number of parameters is significantly larger than the sample size. Existing literature develops a variety of individual tests. Some of them are sensitive to the dense and small disturbance, and others are sensitive to the sparse and large disturbance. Hence, the powers of these tests depend on the assumption of the alternative scenario. This paper provides a unified framework for developing new tests which are adaptive to a large variety of alternative scenarios in high dimensions. In particular, our framework includes arbitrary hypotheses which can be tested using high dimensional $U$-statistic based vectors. Under this framework, we first develop a broad family of tests based on a novel variant of the $L_p$-norm with $p\in \{1,\dots,\infty\}$. We then combine these tests to construct a data-adaptive test that is simultaneously powerful under various alternative scenarios. To obtain the asymptotic distributions of these tests, we utilize the multiplier bootstrap for $U$-statistics. In addition, we consider the computational aspect of the bootstrap method and propose a novel low-cost scheme. We prove the optimality of the proposed tests. Thorough numerical results on simulated and real datasets are provided to support our theory.