Group Inference in High Dimensions with Applications to Hierarchical Testing

TL;DR

This paper introduces a new statistical method for high-dimensional group inference that provides valid confidence intervals and tests, applicable to complex data analysis tasks like hierarchical testing and treatment effect detection.

Contribution

It develops asymptotically unbiased estimators for weighted quadratic functionals and a novel inference procedure with proven asymptotic Gaussian distribution, suitable for large groups.

Findings

Method performs well in simulations and real data applications.

Provides computationally efficient and statistically valid tests for large groups.

Achieves good power for detecting effects in high-dimensional settings.

Abstract

High-dimensional group inference is an essential part of statistical methods for analysing complex data sets, including hierarchical testing, tests of interaction, detection of heterogeneous treatment effects and inference for local heritability. Group inference in regression models can be measured with respect to a weighted quadratic functional of the regression sub-vector corresponding to the group. Asymptotically unbiased estimators of these weighted quadratic functionals are constructed and a novel procedure using these estimators for inference is proposed. We derive its asymptotic Gaussian distribution which enables the construction of asymptotically valid confidence intervals and tests which perform well in terms of length or power. The proposed test is computationally efficient even for a large group, statistically valid for any group size and achieving good power performance for testing large groups with many small regression coefficients. We apply the methodology to several interesting statistical problems and demonstrate its strength and usefulness on simulated and real data.