Structured Transformations for Stable and Interpretable Neural Computation

TL;DR

This paper proposes a new structured transformation approach for neural networks that enhances stability, interpretability, and robustness without sacrificing performance, by decomposing layer transformations into structured linear and residual components.

Contribution

It introduces a reformulation of layer transformations into structured linear and residual parts, promoting stable and interpretable neural computation.

Findings

Improved gradient conditioning and robustness.

Reduced sensitivity to perturbations.

Stable information flow across layers.

Abstract

Despite their impressive performance, contemporary neural networks often lack structural safeguards that promote stable learning and interpretable behavior. In this work, we introduce a reformulation of layer-level transformations that departs from the standard unconstrained affine paradigm. Each transformation is decomposed into a structured linear operator and a residual corrective component, enabling more disciplined signal propagation and improved training dynamics. Our formulation encourages internal consistency and supports stable information flow across depth, while remaining fully compatible with standard learning objectives and backpropagation. Through a series of synthetic and real-world experiments, we demonstrate that models constructed with these structured transformations exhibit improved gradient conditioning, reduced sensitivity to perturbations, and layer-wise robustness. We further show that these benefits persist across architectural scales and training regimes. This study serves as a foundation for a more principled class of neural architectures that prioritize stability and transparency-offering new tools for reasoning about learning behavior without sacrificing expressive power.