A unified framework for correlation mining in ultra-high dimension

TL;DR

This paper introduces a unified framework for correlation and partial correlation mining in ultra-high dimensional data, overcoming previous limitations related to sparsity assumptions and computational challenges, with theoretical insights connecting to random geometric graphs.

Contribution

It proposes a general methodology not restricted to block diagonal structures, linking correlation screening to random geometric graphs, and establishing a duality between correlation and partial correlation screening.

Findings

Finite-sample compound Poisson characterizations for highly correlated variables.

Framework applicable to both finite and infinite dimensional settings.

Demonstrates duality between correlation and partial correlation screening.

Abstract

Many applications benefit from theory relevant to the identification of variables having large correlations or partial correlations in high dimension. Recently there has been progress in the ultra-high dimensional setting when the sample size $n$ is fixed and the dimension $p$ tends to infinity. Despite these advances, the correlation screening framework suffers from practical, methodological and theoretical deficiencies. For instance, previous correlation screening theory requires that the population covariance matrix be sparse and block diagonal. This block sparsity assumption is however restrictive in practical applications. As a second example, correlation and partial correlation screening requires the estimation of dependence measures, which can be computationally prohibitive. In this paper, we propose a unifying approach to correlation and partial correlation mining that is not restricted to block diagonal correlation structure, thus yielding a methodology that is suitable for modern applications. By making connections to random geometric graphs, the number of highly correlated or partial correlated variables are shown to have compound Poisson finite-sample characterizations, which hold for both the finite $p$ case and when $p$ tends to infinity. The unifying framework also demonstrates a duality between correlation and partial correlation screening with theoretical and practical consequences.