Effective Learning of a GMRF Mixture Model

TL;DR

This paper introduces a novel two-step method for learning Gaussian Markov Random Field Mixture Models that effectively estimates sparse precision matrices, overcoming bias issues associated with traditional methods like GLASSO, and demonstrates superior performance on high-dimensional data.

Contribution

The paper proposes a debiasing approach for GMRF mixture models that improves precision matrix estimation and outperforms existing methods like GLASSO in high-dimensional settings.

Findings

Outperforms GLASSO in accuracy of precision matrix estimation.

Yields better results in learning priors for image patches.

Achieves superior performance on synthetic and real datasets.

Abstract

Learning a Gaussian Mixture Model (GMM) is hard when the number of parameters is too large given the amount of available data. As a remedy, we propose restricting the GMM to a Gaussian Markov Random Field Mixture Model (GMRF-MM), as well as a new method for estimating the latter's sparse precision (i.e., inverse covariance) matrices. When the sparsity pattern of each matrix is known, we propose an efficient optimization method for the Maximum Likelihood Estimate (MLE) of that matrix. When it is unknown, we utilize the popular Graphical Least Absolute Shrinkage and Selection Operator (GLASSO) to estimate that pattern. However, we show that even for a single Gaussian, when GLASSO is tuned to successfully estimate the sparsity pattern, it does so at the price of a substantial bias of the values of the nonzero entries of the matrix, and we show that this problem only worsens in a mixture setting. To overcome this, we discard the nonzero values estimated by GLASSO, keep only its pattern estimate and use it within the proposed MLE method. This yields an effective two-step procedure that removes the bias. We show that our "debiasing" approach outperforms GLASSO in both the single-GMRF and the GMRF-MM cases. We also show that when learning priors for image patches, our method outperforms GLASSO even if we merely use an educated guess about the sparsity pattern, and that our GMRF-MM outperforms the baseline GMM on real and synthetic high-dimensional datasets.