Revisiting Orbital Minimization Method for Neural Operator Decomposition

TL;DR

This paper revisits the classical orbital minimization method (OMM) from computational physics, adapting it for neural network decomposition of operators, and demonstrates its effectiveness in modern machine learning tasks.

Contribution

It provides a theoretical justification for OMM's broader use in neural network training and adapts it for decomposing positive semidefinite operators in ML.

Findings

OMM can be effectively adapted for neural network training.

The method shows practical advantages on benchmark tasks.

Revisiting classical methods enhances modern machine learning approaches.

Abstract

Spectral decomposition of linear operators plays a central role in many areas of machine learning and scientific computing. Recent work has explored training neural networks to approximate eigenfunctions of such operators, enabling scalable approaches to representation learning, dynamical systems, and partial differential equations (PDEs). In this paper, we revisit a classical optimization framework from the computational physics literature known as the \emph{orbital minimization method} (OMM), originally proposed in the 1990s for solving eigenvalue problems in computational chemistry. We provide a simple linear-algebraic proof of the consistency of the OMM objective, and reveal connections between this method and several ideas that have appeared independently across different domains. Our primary goal is to justify its broader applicability in modern learning pipelines. We adapt this framework to train neural networks to decompose positive semidefinite operators, and demonstrate its practical advantages across a range of benchmark tasks. Our results highlight how revisiting classical numerical methods through the lens of modern theory and computation can provide not only a principled approach for deploying neural networks in numerical simulation, but also effective and scalable tools for machine learning.