Gaussian mixture layers for neural networks

TL;DR

This paper introduces Gaussian mixture layers for neural networks, deriving training dynamics using Wasserstein gradient flows, and demonstrates their effectiveness and distinct behavior compared to traditional layers in classification tasks.

Contribution

It proposes a novel Gaussian mixture layer architecture with dynamics derived from Wasserstein gradient flows, expanding the mean-field theory approach to finite-width networks.

Findings

GM layers achieve comparable test performance to two-layer networks.

GM layers exhibit different dynamics than classical fully connected layers.

The approach bridges mean-field theory and practical neural network design.

Abstract

The mean-field theory for two-layer neural networks considers infinitely wide networks that are linearly parameterized by a probability measure over the parameter space. This nonparametric perspective has significantly advanced both the theoretical and conceptual understanding of neural networks, with substantial efforts made to validate its applicability to networks of moderate width. In this work, we explore the opposite direction, investigating whether dynamics can be directly implemented over probability measures. Specifically, we employ Gaussian mixture models as a flexible and expressive parametric family of distributions together with the theory of Wasserstein gradient flows to derive training dynamics for such measures. Our approach introduces a new type of layer -- the Gaussian mixture (GM) layer -- that can be integrated into neural network architectures. As a proof of concept, we validate our proposal through experiments on simple classification tasks, where a GM layer achieves test performance comparable to that of a two-layer fully connected network. Furthermore, we examine the behavior of these dynamics and demonstrate numerically that GM layers exhibit markedly different behavior compared to classical fully connected layers, even when the latter are large enough to be considered in the mean-field regime.