Spiky Rank and Its Applications to Rigidity and Circuits

TL;DR

This paper introduces spiky rank, a new matrix complexity measure combining combinatorial and algebraic features, with applications to matrix rigidity and neural network circuit lower bounds.

Contribution

It defines spiky rank, explores its properties, and demonstrates its usefulness in deriving lower bounds for matrix rigidity and neural network circuits.

Findings

Large spiky rank implies high matrix rigidity.

Spiky rank lower bounds lead to neural network circuit complexity bounds.

Developed bounds for random matrices and applications to expanders.

Abstract

We introduce spiky rank, a new matrix parameter that enhances blocky rank by combining the combinatorial structure of the latter with linear-algebraic flexibility. A spiky matrix is block-structured with diagonal blocks that are arbitrary rank-one matrices, and the spiky rank of a matrix is the minimum number of such matrices required to express it as a sum. This measure extends blocky rank to real matrices and is more robust for problems with both combinatorial and algebraic character. Our conceptual contribution is as follows: we propose spiky rank as a well-behaved candidate matrix complexity measure and demonstrate its potential through applications. We show that large spiky rank implies high matrix rigidity, and that spiky rank lower bounds yield lower bounds for depth-2 ReLU circuits, the basic building blocks of neural networks. On the technical side, we establish tight bounds for random matrices and develop a framework for explicit lower bounds, applying it to Hamming distance matrices and spectral expanders. Finally, we relate spiky rank to other matrix parameters, including blocky rank, sparsity, and the $\gamma_2$-norm.