S$^3$: Structured Sparsity Specification

TL;DR

S$^3$ is an algebraic framework that precisely defines, composes, and implements diverse structured sparsity patterns in neural networks, enabling improved pruning strategies.

Contribution

It introduces a formal, composable framework for structured sparsity specification that integrates with existing pruning methods and demonstrates superior experimental results.

Findings

S$^3$ can specify a wide range of sparsity patterns including N:M and channel pruning.

Structured OBS and OBD implementations based on S$^3$ outperform second-order heuristics.

The framework is mathematically formalized and validated through experiments.

Abstract

We introduce the Structured Sparsity Specification (S$^3$), an algebraic framework for defining, composing, and implementing structured sparse patterns. S$^3$ specifies sparsity through three components: a View that reshapes the tensor via layout composition, a Block specification that defines the atomic pruning unit, and the sparsity decision Scope. Both Block and Scope support Coupling across tensors for coordinated sparsification. S$^3$ enables precise specification of diverse sparsity structures, from fine-grained N:M patterns to coarse channel pruning, and integrates seamlessly with Optimal Brain Damage (OBD) and Surgeon (OBS). We formalize the framework mathematically, demonstrate its expressiveness on canonical patterns, and validate it experimentally via structured OBS and OBD implementations built entirely on S$^3$, which surpasses well-established second order heuristics on output reconstruction across common configurations.