The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information

TL;DR

This paper introduces an iterative optimal brain surgeon method that leverages second-order curvature information to improve sparse recovery in deep neural networks, providing theoretical guarantees and demonstrating strong empirical results on large-scale models.

Contribution

It develops new sparse recovery algorithms inspired by OBS with theoretical guarantees and applies them effectively to prune large-scale Transformer models.

Findings

Improved convergence bounds for sparse recovery algorithms.

Enhanced pruning performance on large-scale vision and language models.

Theoretical analysis connecting OBS and sparse recovery methods.

Abstract

The rising footprint of machine learning has led to a focus on imposing \emph{model sparsity} as a means of reducing computational and memory costs. For deep neural networks (DNNs), the state-of-the-art accuracy-vs-sparsity is achieved by heuristics inspired by the classical Optimal Brain Surgeon (OBS) framework~\citep{lecun90brain, hassibi1992second, hassibi1993optimal}, which leverages loss curvature information to make better pruning decisions. Yet, these results still lack a solid theoretical understanding, and it is unclear whether they can be improved by leveraging connections to the wealth of work on sparse recovery algorithms. In this paper, we draw new connections between these two areas and present new sparse recovery algorithms inspired by the OBS framework that comes with theoretical guarantees under reasonable assumptions and have strong practical performance. Specifically, our work starts from the observation that we can leverage curvature information in OBS-like fashion upon the projection step of classic iterative sparse recovery algorithms such as IHT. We show for the first time that this leads both to improved convergence bounds under standard assumptions. Furthermore, we present extensions of this approach to the practical task of obtaining accurate sparse DNNs, and validate it experimentally at scale for Transformer-based models on vision and language tasks.