Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networks

TL;DR

This paper introduces a Structural Correspondence framework demonstrating that Diagonal plus Low-Rank neural networks can achieve universal approximation, overcoming limitations of purely low-rank models.

Contribution

It proves that augmenting low-rank layers with a minimal diagonal component suffices for universal approximation, without relying on dense matrices or restrictive activation functions.

Findings

DLoR structures can exactly reconstruct full-rank transformations.

DLoR networks restore the Universal Approximation Theorem for general activations.

Multiplicative depth offers better parameter-to-expressivity scaling than additive width.

Abstract

The massive computational costs of scaling modern deep learning architectures have driven the widespread use of parameter-efficient low-rank structures, such as LoRA and low-rank factorization. However, theoretical guarantees for their expressive power are less explored, often relying on restrictive priors like a pretrained base matrix, ReLU activations or non-verifiable singularity conditions. We first investigate the limits of neural networks constrained strictly to low-rank manifolds without pretrained dense priors. We demonstrate a theoretical paradox: while purely rank-1 layers can exactly interpolate arbitrary scalar datasets, they collapse for function approximations. To overcome this bottleneck without surrendering parameter efficiency, we introduce a unified \textit{Structural Correspondence} framework. We prove that augmenting low-rank layers with only a minimal sparse diagonal component, say a Diagonal plus Low-Rank (DLoR) structure, is sufficient to reach Universal Approximation. We show that any full-rank transformation can be exactly reconstructed using these DLoR components by trading off network width (additive decomposition) or depth (multiplicative decomposition). By tracking asymptotic Taylor remainders, we prove that DLoR neural networks fully restore the Universal Approximation Theorem for general activation functions. Finally, we establish that multiplicative depth provides superior parameter-to-expressivity scaling compared to additive width. Our results show that dense matrices and specific activation functions are not topological prerequisites for universal expressivity.