Unit-Consistent (UC) Adjoint for GSD and Backprop in Deep Learning Applications

TL;DR

This paper introduces a novel unit-consistent adjoint method for backpropagation in deep neural networks, ensuring gauge invariance and improving optimization consistency across parameterizations.

Contribution

It proposes a new operator-level UC adjoint for backpropagation, enhancing gauge invariance in deep learning optimization.

Findings

UC adjoint improves parameterization invariance

Method applies uniformly across network components

Enhances stability of gradient-based optimization

Abstract

Deep neural networks constructed from linear maps and positively homogeneous nonlinearities (e.g., ReLU) possess a fundamental gauge symmetry: the network function is invariant to node-wise diagonal rescalings. However, standard gradient descent is not equivariant to this symmetry, causing optimization trajectories to depend heavily on arbitrary parameterizations. Prior work has proposed rescaling-invariant optimization schemes for positively homogeneous networks (e.g., path-based or path-space updates). Our contribution is complementary: we formulate the invariance requirement at the level of the backward adjoint/optimization geometry, which provides a simple, operator-level recipe that can be applied uniformly across network components and optimizer state. By replacing the Euclidean transpose with a Unit-Consistent (UC) adjoint, we derive UC gauge-consistent steepest descent and backprogation.