Diagonally-Weighted Generalized Method of Moments Estimation for Gaussian Mixture Modeling

TL;DR

This paper introduces a diagonally-weighted GMM method that improves efficiency and stability for high-dimensional Gaussian mixture estimation, outperforming traditional MM and GMM in accuracy and runtime.

Contribution

The paper proposes DGMM, a novel diagonally-weighted GMM approach that reduces computational and storage complexity while maintaining statistical efficiency for high-dimensional data.

Findings

DGMM achieves smaller estimation errors than MM and GMM.

DGMM requires significantly less runtime than traditional methods.

Empirical results validate DGMM's efficiency and stability.

Abstract

Since Pearson [Philosophical Transactions of the Royal Society of London. A, 185 (1894), pp. 71-110] first applied the method of moments (MM) for modeling data as a mixture of one-dimensional Gaussians, moment-based estimation methods have proliferated. Among these methods, the generalized method of moments (GMM) improves the statistical efficiency of MM by weighting the moments appropriately. However, the computational complexity and storage complexity of MM and GMM grow exponentially with the dimension, making these methods impractical for high-dimensional data or when higher-order moments are required. Such computational bottlenecks are more severe in GMM since it additionally requires estimating a large weighting matrix. To overcome these bottlenecks, we propose the diagonally-weighted GMM (DGMM), which achieves a balance among statistical efficiency, computational complexity, and numerical stability. We apply DGMM to study the parameter estimation problem for weakly separated heteroscedastic low-rank Gaussian mixtures and design a computationally efficient and numerically stable algorithm that obtains the DGMM estimator without explicitly computing or storing the moment tensors. We implement the proposed algorithm and empirically validate the advantages of DGMM: in numerical studies, DGMM attains smaller estimation errors while requiring substantially shorter runtime than MM and GMM. The code and data will be available upon publication at https://github.com/liu-lzhang/dgmm.