Learning to Discover Iterative Spectral Algorithms

TL;DR

AutoSpec is a neural network framework that autonomously discovers iterative spectral algorithms, improving large-scale linear algebra and optimization tasks by adapting to spectral properties and outperforming basic baselines.

Contribution

Introduces AutoSpec, a self-supervised neural network approach for discovering spectral algorithms tailored to specific tasks and spectral profiles.

Findings

Orders-of-magnitude improvements in accuracy and iteration count on real-world matrices.

Discovered algorithms exhibit classical Chebyshev polynomial behavior.

Effective transfer from synthetic to real-world large-scale problems.

Abstract

We introduce AutoSpec, a neural network framework for discovering iterative spectral algorithms for large-scale numerical linear algebra and numerical optimization. Our self-supervised models adapt to input operators using coarse spectral information (e.g., eigenvalue estimates and residual norms), and they predict recurrence coefficients for computing or applying a matrix polynomial tailored to a downstream task. The effectiveness of AutoSpec relies on three ingredients: an architecture whose inference pass implements short, executable numerical linear algebra recurrences; efficient training on small synthetic problems with transfer to large-scale real-world operators; and task-defined objectives that enforce the desired approximation or preconditioning behavior across the range of spectral profiles represented in the training set. We apply AutoSpec to discovering algorithms for representative numerical linear algebra tasks: accelerating matrix-function approximation; accelerating sparse linear solvers; and spectral filtering/preconditioning for eigenvalue computations. On real-world matrices, the learned procedures deliver orders-of-magnitude improvements in accuracy and/or reductions in iteration count, relative to basic baselines. We also find clear connections to classical theory: the induced polynomials often exhibit near-equiripple, near-minimax behavior characteristic of Chebyshev polynomials.