Phase transition for conditional covariance matrices estimated by importance sampling, and implications for cross-entropy schemes in high dimension

TL;DR

This paper investigates a phase transition phenomenon in high-dimensional covariance matrix estimation via importance sampling, revealing that the smallest eigenvalue significantly influences the success of cross-entropy algorithms for rare event probability estimation.

Contribution

It introduces a phase transition analysis for covariance matrix estimation in high dimensions, linking the success of importance sampling to spectral properties like the smallest eigenvalue.

Findings

A phase transition occurs at a critical sample size exponent $\kappa_*$.

Importance sampling performs better with larger smallest eigenvalues.

Numerical simulations confirm the spectral influence on CE scheme efficiency.

Abstract

Motivated by the estimation of covariance matrices by importance sampling arising in the cross-entropy (CE) algorithm, we study a random matrix model $\hat \Sigma = {\bf X} L {\bf X}^\top$ with two distinct features: $\bf X$ and $L$ are dependent, and $L$ is heavy-tailed. In the high-dimensional regime $d \to \infty$, we prove under suitable assumptions that a phase transition occurs in the polynomial regime $n = d^\kappa$, with $n$ the sample size. Namely, we prove that $\lVert \hat \Sigma - E \hat \Sigma \rVert \Rightarrow 0$ if and only if $\kappa > \kappa_*$ for some threshold $\kappa_*$ determined by the behavior of the maximum likelihood ratios. Moreover, we identify general situations where $\kappa_* = 1/\lambda_1$, with $\lambda_1$ the smallest eigenvalue of the covariance matrix of the auxiliary distribution used to estimate $\hat \Sigma$ by importance sampling. This suggests that importance sampling will work better with covariance matrices having a large smallest eigenvalue. We carry this insight into recent CE schemes proposed to estimate the probability of high-dimensional rare events. Through numerical simulations, we demonstrate that better CE schemes are also the ones with larger smallest eigenvalue, even though these algorithms were not designed to smooth the spectrum. This new spectral interpretation raises stimulating questions and opens research directions for the design of efficient high-dimensional algorithms.