Gaussian Mixture Models with Component Means Constrained in Pre-selected Subspaces

TL;DR

This paper introduces a constrained Gaussian mixture model with component means in a pre-selected subspace, using an EM algorithm and kernel density methods to improve classification and clustering, especially for visualization.

Contribution

It proposes a novel subspace-constrained GMM estimation approach using kernel density modes and class means, enhancing classification and clustering performance.

Findings

The method outperforms reduced rank MDA in low-dimensional settings.

Multiple subspaces are derived from kernel densities with varying bandwidths.

The approach is effective for visualization and dimension reduction.

Abstract

We investigate a Gaussian mixture model (GMM) with component means constrained in a pre-selected subspace. Applications to classification and clustering are explored. An EM-type estimation algorithm is derived. We prove that the subspace containing the component means of a GMM with a common covariance matrix also contains the modes of the density and the class means. This motivates us to find a subspace by applying weighted principal component analysis to the modes of a kernel density and the class means. To circumvent the difficulty of deciding the kernel bandwidth, we acquire multiple subspaces from the kernel densities based on a sequence of bandwidths. The GMM constrained by each subspace is estimated; and the model yielding the maximum likelihood is chosen. A dimension reduction property is proved in the sense of being informative for classification or clustering. Experiments on real and simulated data sets are conducted to examine several ways of determining the subspace and to compare with the reduced rank mixture discriminant analysis (MDA). Our new method with the simple technique of spanning the subspace only by class means often outperforms the reduced rank MDA when the subspace dimension is very low, making it particularly appealing for visualization.