A Discriminative Gaussian Mixture Model with Sparsity

TL;DR

This paper introduces a sparse discriminative Gaussian mixture model (SDGM) that enhances classification by addressing unimodality limitations, reducing model complexity, and improving generalization, with successful integration into neural networks.

Contribution

The paper presents a novel SDGM that employs sparse Bayesian learning for efficient, end-to-end training and better classification performance compared to traditional softmax models.

Findings

Outperforms existing softmax-based discriminative models

Reduces model complexity through sparsity

Can be integrated into neural networks for end-to-end training

Abstract

In probabilistic classification, a discriminative model based on the softmax function has a potential limitation in that it assumes unimodality for each class in the feature space. The mixture model can address this issue, although it leads to an increase in the number of parameters. We propose a sparse classifier based on a discriminative GMM, referred to as a sparse discriminative Gaussian mixture (SDGM). In the SDGM, a GMM-based discriminative model is trained via sparse Bayesian learning. Using this sparse learning framework, we can simultaneously remove redundant Gaussian components and reduce the number of parameters used in the remaining components during learning; this learning method reduces the model complexity, thereby improving the generalization capability. Furthermore, the SDGM can be embedded into neural networks (NNs), such as convolutional NNs, and can be trained in an end-to-end manner. Experimental results demonstrated that the proposed method outperformed the existing softmax-based discriminative models.