On the Influence of Enforcing Model Identifiability on Learning dynamics of Gaussian Mixture Models

TL;DR

This paper introduces a reparameterization technique that enforces identifiability in singular models like GMMs, leading to improved learning dynamics, faster convergence, and better understanding of model behavior.

Contribution

The authors propose a general reparameterization method to extract regular submodels from singular models, enhancing training stability and interpretability.

Findings

Faster convergence of gradient descent and EM algorithms for GMMs.

Improved manifold shape around singularities.

Potential applicability to deep neural networks.

Abstract

A common way to learn and analyze statistical models is to consider operations in the model parameter space. But what happens if we optimize in the parameter space and there is no one-to-one mapping between the parameter space and the underlying statistical model space? Such cases frequently occur for hierarchical models which include statistical mixtures or stochastic neural networks, and these models are said to be singular. Singular models reveal several important and well-studied problems in machine learning like the decrease in convergence speed of learning trajectories due to attractor behaviors. In this work, we propose a relative reparameterization technique of the parameter space, which yields a general method for extracting regular submodels from singular models. Our method enforces model identifiability during training and we study the learning dynamics for gradient descent and expectation maximization for Gaussian Mixture Models (GMMs) under relative parameterization, showing faster experimental convergence and a improved manifold shape of the dynamics around the singularity. Extending the analysis beyond GMMs, we furthermore analyze the Fisher information matrix under relative reparameterization and its influence on the generalization error, and show how the method can be applied to more complex models like deep neural networks.