Towards Modeling and Resolving Singular Parameter Spaces using Stratifolds

TL;DR

This paper introduces a novel method using stratifolds from algebraic topology to model and resolve singularities in parameter spaces, improving convergence in statistical models and neural networks.

Contribution

It proposes a general approach to model singular parameter spaces with stratifolds and constructs smooth manifold approximations to enhance learning dynamics.

Findings

Using manifold approximation speeds up convergence.

Stratifold-based modeling avoids singular attractors.

Empirical results show improved learning efficiency.

Abstract

When analyzing parametric statistical models, a useful approach consists in modeling geometrically the parameter space. However, even for very simple and commonly used hierarchical models like statistical mixtures or stochastic deep neural networks, the smoothness assumption of manifolds is violated at singular points which exhibit non-smooth neighborhoods in the parameter space. These singular models have been analyzed in the context of learning dynamics, where singularities can act as attractors on the learning trajectory and, therefore, negatively influence the convergence speed of models. We propose a general approach to circumvent the problem arising from singularities by using stratifolds, a concept from algebraic topology, to formally model singular parameter spaces. We use the property that specific stratifolds are equipped with a resolution method to construct a smooth manifold approximation of the singular space. We empirically show that using (natural) gradient descent on the smooth manifold approximation instead of the singular space allows us to avoid the attractor behavior and therefore improve the convergence speed in learning.