Adaptive Testing for High-dimensional Data

TL;DR

This paper introduces a flexible class of $L_q$-norm based U-statistics for high-dimensional global testing problems, offering asymptotic properties, computational efficiency, and improved power against various alternatives.

Contribution

It extends previous work by providing a general framework for $L_q$-based tests, including a computationally efficient variant and a combined testing procedure with high power.

Findings

Asymptotic normality of $L_q$-based U-statistics under mild conditions.

A combined test achieves high power across different sparsity levels.

Proposed methods outperform existing tests in numerical simulations.

Abstract

In this article, we propose a class of $L_q$-norm based U-statistics for a family of global testing problems related to high-dimensional data. This includes testing of mean vector and its spatial sign, simultaneous testing of linear model coefficients, and testing of component-wise independence for high-dimensional observations, among others. Under the null hypothesis, we derive asymptotic normality and independence between $L_q$-norm based U-statistics for several $q$s under mild moment and cumulant conditions. A simple combination of two studentized $L_q$-based test statistics via their $p$-values is proposed and is shown to attain great power against alternatives of different sparsity. Our work is a substantial extension of He et al. (2021), which is mostly focused on mean and covariance testing, and we manage to provide a general treatment of asymptotic independence of $L_q$-norm based U-statistics for a wide class of kernels. To alleviate the computation burden, we introduce a variant of the proposed U-statistics by using the monotone indices in the summation, resulting in a U-statistic with asymmetric kernel. A dynamic programming method is introduced to reduce the computational cost from $O(n^{qr})$, which is required for the calculation of the full U-statistic, to $O(n^r)$ where $r$ is the order of the kernel. Numerical studies further corroborate the advantage of the proposed adaptive test as compared to some existing competitors.