Large-scale gradient-based training of Mixtures of Factor Analyzers

TL;DR

This paper introduces a scalable, gradient-based training method for Mixtures of Factor Analyzers (MFA), enabling efficient high-dimensional data modeling, with theoretical insights and practical applications to image datasets.

Contribution

It presents a new stochastic gradient descent approach for MFA training, simplifying initialization and avoiding batch algorithm issues, with theoretical analysis on matrix computations.

Findings

Effective MFA training on high-dimensional data

Sample generation and outlier detection demonstrated

Training requires only small matrix inversions

Abstract

Gaussian Mixture Models (GMMs) are a standard tool in data analysis. However, they face problems when applied to high-dimensional data (e.g., images) due to the size of the required full covariance matrices (CMs), whereas the use of diagonal or spherical CMs often imposes restrictions that are too severe. The Mixture of Factor analyzers (MFA) model is an important extension of GMMs, which allows to smoothly interpolate between diagonal and full CMs based on the number of \textit{factor loadings} $l$. MFA has successfully been applied for modeling high-dimensional image data. This article contributes both a theoretical analysis as well as a new method for efficient high-dimensional MFA training by stochastic gradient descent, starting from random centroid initializations. This greatly simplifies the training and initialization process, and avoids problems of batch-type algorithms such Expectation-Maximization (EM) when training with huge amounts of data. In addition, by exploiting the properties of the matrix determinant lemma, we prove that MFA training and inference/sampling can be performed based on precision matrices, which does not require matrix inversions after training is completed. At training time, the methods requires the inversion of $l\times l$ matrices only. Besides the theoretical analysis and proofs, we apply MFA to typical image datasets such as SVHN and MNIST, and demonstrate the ability to perform sample generation and outlier detection.