Scale redundancy and soft gauge fixing in positively homogeneous neural networks

TL;DR

This paper explores the gauge symmetry in positively homogeneous neural networks, introducing gauge-adapted coordinates and a soft orbit-selection method to improve training stability and reduce scale drift.

Contribution

It introduces gauge-adapted coordinates and a soft orbit-selection functional to manage scale redundancy in neural networks, linking gauge theory concepts with optimization.

Findings

Expands stable learning-rate regime

Suppresses scale drift during training

Maintains network expressivity

Abstract

Neural networks with positively homogeneous activations exhibit an exact continuous reparametrization symmetry: neuron-wise rescalings generate parameter-space orbits along which the input--output function is invariant. We interpret this symmetry as a gauge redundancy and introduce gauge-adapted coordinates that separate invariant and scale-imbalance directions. Inspired by gauge fixing in field theory, we introduce a soft orbit-selection (norm-balancing) functional acting only on redundant scale coordinates. We show analytically that it induces dissipative relaxation of imbalance modes to preserve the realized function. In controlled experiments, this orbit-selection penalty expands the stable learning-rate regime and suppresses scale drift without changing expressivity. These results establish a structural link between gauge-orbit geometry and optimization conditioning, providing a concrete connection between gauge-theoretic concepts and machine learning.