Boundary Mass and the Soft-to-Hard Limit in Mixture-of-Experts

TL;DR

This paper analyzes the transition from soft to hard routing in mixture-of-experts models, revealing that the limit is governed by a thin geometric layer around routing interfaces, with implications for model behavior and recovery.

Contribution

It provides a geometric and probabilistic analysis of the soft-to-hard limit in MoE models, including risk bounds and landscape transfer principles.

Findings

Boundary mass is linear in the slab width, controlled by a surface integral.

Soft routing inherits properties of the hard-routing problem under certain conditions.

The zero-temperature limit is governed by a boundary layer, not the entire input space.

Abstract

Softmax-routed mixture-of-experts models approach hard routing as the temperature tends to zero, but this limit is singular near routing ties. This paper studies that singularity at the population level for squared-loss MoE regression. The central object is the \emph{boundary mass}, namely the probability that the top two router scores are separated by only a small margin. Under smoothness and transversality assumptions on the router and input law, we prove coarea/tube estimates showing that this mass is linear in the slab width, with leading constant given by a surface integral over the routing interface in the binary case. These estimates yield quantitative soft-to-hard risk bounds and, under compactness and uniform margin control, $\Gamma$-convergence of the soft objectives to the hard-routing objective. The main conclusion is that the zero-temperature limit is controlled by a thin geometric layer around routing interfaces, not by the full input space. We then use this geometric core in two more model-dependent directions. In a teacher--student setting, we prove a conditional landscape-transfer principle showing that, when the profiled hard-routing problem has favorable identifiability and curvature and the relevant derivatives transfer at boundary-layer scale, small-temperature soft routing inherits approximate teacher recovery and strict-saddle behavior away from teacher-equivalent partitions. We also give a reduced two-expert Gaussian calculation that illustrates a local symmetry-breaking mechanism aligned with the teacher separator.