High-Dimensional Covariance Decomposition into Sparse Markov and Independence Models

TL;DR

This paper introduces a novel high-dimensional covariance decomposition framework that separates data into sparse Markov and independence models, improving model fidelity and inference accuracy.

Contribution

It proposes a new decomposition method combining sparse covariance and precision estimation, with theoretical guarantees and practical validation.

Findings

Consistent support recovery when sample size scales as n = Ω(d^2 log p)

Better inference accuracy demonstrated in experiments

Framework generalizes existing sparse models

Abstract

Fitting high-dimensional data involves a delicate tradeoff between faithful representation and the use of sparse models. Too often, sparsity assumptions on the fitted model are too restrictive to provide a faithful representation of the observed data. In this paper, we present a novel framework incorporating sparsity in different domains.We decompose the observed covariance matrix into a sparse Gaussian Markov model (with a sparse precision matrix) and a sparse independence model (with a sparse covariance matrix). Our framework incorporates sparse covariance and sparse precision estimation as special cases and thus introduces a richer class of high-dimensional models. We characterize sufficient conditions for identifiability of the two models, \viz Markov and independence models. We propose an efficient decomposition method based on a modification of the popular $\ell_1$-penalized maximum-likelihood estimator ($\ell_1$-MLE). We establish that our estimator is consistent in both the domains, i.e., it successfully recovers the supports of both Markov and independence models, when the number of samples $n$ scales as $n = \Omega(d^2 \log p)$, where $p$ is the number of variables and $d$ is the maximum node degree in the Markov model. Our experiments validate these results and also demonstrate that our models have better inference accuracy under simple algorithms such as loopy belief propagation.