A Generalized Singular Value Theory for Neural Networks

TL;DR

This paper extends the generalized singular value decomposition theory to neural networks, showing they can be represented as left-invertible with norm-preserving properties, enabling new analysis and applications.

Contribution

It proves most neural architectures admit a generalized SVD representation with norm-preserving properties and provides algorithms and architectures to estimate and utilize this representation.

Findings

Neural networks can be represented as left-invertible in a generalized SVD form.

The left-invertible nonlinear part can be made norm-preserving, linking input and feature space distances.

The learned representation can identify adversarial perturbations.

Abstract

Building on the abstract Generalized Singular Value Decomposition (GSVD) theory of Brown et al. [2025], we prove that most modern neural architectures admit a generalized SVD representation in which they are left-invertible before a final linear layer, with no change in input-output behavior. Furthermore, the left-invertible nonlinear portion of the input-output behavior can be made to be \emph{norm preserving}, meaning that perturbations in the left-invertible ``embedding'' (the activations prior to the final linear layer in this representation) correspond proportionally to changes in the input, i.e., distance in feature space can be calibrated directly to distance in input space. We provide a data-driven algorithm for estimating this representation from trained models and propose a model architecture that naturally facilitates the decomposition. We then provide a proof-of-concept that the learned representation can be used to identify adversarial perturbations to model inputs, and develop the theory necessary for future applications to areas such as model bias and invertibility.