Estimating The Proportion of Signal Variables Under Arbitrary Covariance Dependence

TL;DR

This paper develops and evaluates estimators for the proportion of signal variables in high-dimensional data with arbitrary covariance dependence, revealing the impact of dependence on estimator performance and proposing a new adaptive estimator.

Contribution

It introduces a new estimator that adapts to arbitrary covariance dependence and provides a comprehensive analysis of how dependence affects estimator performance.

Findings

No single estimator is optimal across all dependence levels.

The new estimator outperforms existing methods in various dependence scenarios.

Dependence level significantly influences the accuracy of signal proportion estimation.

Abstract

Estimating the proportion of signals hidden in a large amount of noise variables is of interest in many scientific inquires. In this paper, we consider realistic but theoretically challenging settings with arbitrary covariance dependence between variables. We define mean absolute correlation (MAC) to measure the overall dependence level and investigate a family of estimators for their performances in the full range of MAC. We explicit the joint effect of MAC dependence and signal sparsity on the performances of the family of estimators and discover that no single estimator in the family is most powerful under different MAC dependence levels. Informed by the theoretical insight, we propose a new estimator to better adapt to arbitrary covariance dependence. The proposed method compares favorably to several existing methods in extensive finite-sample settings with strong to weak covariance dependence and real dependence structures from genetic association studies.