Learning Orthonormal Bases for Function Spaces

TL;DR

This paper introduces a neural network-based method to adapt and optimize infinite-dimensional orthonormal bases in function spaces by modeling paths on the Lie manifold of the orthogonal group via ODEs.

Contribution

It proposes a novel framework to represent and optimize orthonormal bases using neural networks and ODEs on the Lie manifold, with a universality proof for finite-rank generators.

Findings

Able to transform Fourier basis into principal components of datasets

Demonstrated basis adaptation to eigenfunctions of linear operators

Showed flexibility in modeling dynamic modes in physical simulations

Abstract

Infinite-dimensional orthonormal basis expansions play a central role in representing and computing with function spaces due to their favorable linear algebraic properties. However, common bases such as Fourier or wavelets are fixed and do not adapt to the structure of a given problem or dataset. In this paper, we aim to represent these bases with neural networks and optimize them. Our key idea is that any target infinite-dimensional orthonormal basis can be viewed either as a point on the Lie manifold of the orthogonal group, or equivalently, as the endpoint of a continuous path on that manifold that connects a reference basis, e.g. Fourier, to that target. Paths on the Lie manifold satisfy ordinary differential equations (ODEs) governed by skew-adjoint integral operators. Using neural networks to define finite-rank generators of such ODEs allows us to parameterize and optimize orthonormal bases in function space. While relying on finite-rank generators to model infinite operators might seem restrictive, we prove a universality result: even with a rank-2 generator, the integrated solutions of the ODE are dense in the orthogonal group under the appropriate operator topology. In other words, for any target orthonormal basis, there exists a path originating from a reference basis and driven by finite-rank generators that gets arbitrarily close to that target basis. We demonstrate the flexibility of our framework by transforming the Fourier basis into the principal components of a functional dataset, eigenfunctions of linear operators, or dynamic modes of energy-preserving physical simulations.