A Tighter Complexity Analysis of SparseGPT

TL;DR

This paper refines the complexity analysis of SparseGPT, reducing its theoretical running time from cubic to approximately quadratic, by leveraging advanced matrix multiplication exponents and analyzing lazy update behaviors.

Contribution

It provides a tighter, more precise complexity bound for SparseGPT's running time, improving upon previous analyses and applying to iterative maintenance problems.

Findings

Reduced SparseGPT complexity from O(d^3) to O(d^{2.53}) with current matrix multiplication exponents.

Introduced a generalized analysis framework for lazy update behaviors in iterative algorithms.

Demonstrated the applicability of the analysis to related problems in matrix maintenance.

Abstract

In this work, we improved the analysis of the running time of SparseGPT [Frantar, Alistarh ICML 2023] from $O(d^{3})$ to $O(d^{\omega} + d^{2+a+o(1)} + d^{1+\omega(1,1,a)-a})$ for any $a \in [0, 1]$, where $\omega$ is the exponent of matrix multiplication. In particular, for the current $\omega \approx 2.371$ [Alman, Duan, Williams, Xu, Xu, Zhou 2024], our running time boils down to $O(d^{2.53})$. This running time is due to the analysis of the lazy update behavior in iterative maintenance problems such as [Deng, Song, Weinstein 2022; Brand, Song, Zhou ICML 2024].