Zeroth-Order Topological Insights into Iterative Magnitude Pruning

TL;DR

This paper uses topological data analysis to explain why iterative magnitude pruning effectively retains essential network weights, providing theoretical bounds and a modified pruning method to preserve topological features.

Contribution

It introduces a topological perspective to understand IMP, offering bounds on pruning while maintaining topological features and proposing a modified IMP method.

Findings

IMP encourages retention of weights preserving topological information

Bounds on pruning while preserving zeroth order topological features

A modified IMP that better preserves topological structure

Abstract

Modern-day neural networks are famously large, yet also highly redundant and compressible; there exist numerous pruning strategies in the deep learning literature that yield over 90% sparser sub-networks of fully-trained, dense architectures while still maintaining their original accuracies. Amongst these many methods though -- thanks to its conceptual simplicity, ease of implementation, and efficacy -- Iterative Magnitude Pruning (IMP) dominates in practice and is the de facto baseline to beat in the pruning community. However, theoretical explanations as to why a simplistic method such as IMP works at all are few and limited. In this work, we leverage the notion of persistent homology to gain insights into the workings of IMP and show that it inherently encourages retention of those weights which preserve topological information in a trained network. Subsequently, we also provide bounds on how much different networks can be pruned while perfectly preserving their zeroth order topological features, and present a modified version of IMP to do the same.