Eigen Neural Network: Unlocking Generalizable Vision with Eigenbasis

TL;DR

The Eigen Neural Network (ENN) introduces a novel weight reparameterization using an orthonormal eigenbasis, resulting in more structured features, improved performance on large-scale benchmarks, and a highly efficient, backpropagation-free training variant.

Contribution

ENN is the first architecture to enforce decorrelated, well-aligned weights via a learned eigenbasis, improving feature representations and enabling a parallelizable, BP-free training method.

Findings

Outperforms state-of-the-art on ImageNet classification

Sets new benchmark in cross-modal image-text retrieval

Achieves over 2x training speedup with ENN-ℓ

Abstract

The remarkable success of Deep Neural Networks(DNN) is driven by gradient-based optimization, yet this process is often undermined by its tendency to produce disordered weight structures, which harms feature clarity and degrades learning dynamics. To address this fundamental representational flaw, we introduced the Eigen Neural Network (ENN), a novel architecture that reparameterizes each layer's weights in a layer-shared, learned orthonormal eigenbasis. This design enforces decorrelated, well-aligned weight dynamics axiomatically, rather than through regularization, leading to more structured and discriminative feature representations. When integrated with standard BP, ENN consistently outperforms state-of-the-art methods on large-scale image classification benchmarks, including ImageNet, and its superior representations generalize to set a new benchmark in cross-modal image-text retrieval. Furthermore, ENN's principled structure enables a highly efficient, backpropagation-free(BP-free) local learning variant, ENN-$\ell$. This variant not only resolves BP's procedural bottlenecks to achieve over 2$\times$ training speedup via parallelism, but also, remarkably, surpasses the accuracy of end-to-end backpropagation. ENN thus presents a new architectural paradigm that directly remedies the representational deficiencies of BP, leading to enhanced performance and enabling a more efficient, parallelizable training regime.