Position: Curvature Matrices Should Be Democratized via Linear Operators

TL;DR

This paper advocates for using linear operators as a unified, scalable interface to handle complex curvature matrices in machine learning, simplifying computations and enabling large-scale applications.

Contribution

It introduces the curvlinops library, which provides a unified linear operator interface for curvature matrices, improving scalability, usability, and extensibility.

Findings

The linear operator interface simplifies curvature matrix computations.

curvlinops enables scalable second-order optimization and analysis.

The approach improves interoperability with existing machine learning tools.

Abstract

Structured large matrices are prevalent in machine learning. A particularly important class is curvature matrices like the Hessian, which are central to understanding the loss landscape of neural nets (NNs), and enable second-order optimization, uncertainty quantification, model pruning, data attribution, and more. However, curvature computations can be challenging due to the complexity of automatic differentiation, and the variety and structural assumptions of curvature proxies, like sparsity and Kronecker factorization. In this position paper, we argue that linear operators -- an interface for performing matrix-vector products -- provide a general, scalable, and user-friendly abstraction to handle curvature matrices. To support this position, we developed $\textit{curvlinops}$, a library that provides curvature matrices through a unified linear operator interface. We demonstrate with $\textit{curvlinops}$ how this interface can hide complexity, simplify applications, be extensible and interoperable with other libraries, and scale to large NNs.