Mask in the Mirror: Implicit Sparsification

TL;DR

This paper provides a theoretical analysis of continuous sparsification in neural networks, revealing an implicit transition from L2 to L1 regularization, and introduces PILoT, a method that dynamically controls this bias to improve sparsification performance.

Contribution

It offers a theoretical explanation for implicit regularization dynamics and proposes PILoT, a novel sparsification method with dynamic regularization and improved results.

Findings

Implicit sparsification transitions from L2 to L1 regularization over time.

PILoT outperforms baseline methods in standard experiments.

Theoretical guarantees for convergence and optimality are established.

Abstract

Continuous sparsification strategies are among the most effective methods for reducing the inference costs and memory demands of large-scale neural networks. A key factor in their success is the implicit $L_1$ regularization induced by jointly learning both mask and weight variables, which has been shown experimentally to outperform explicit $L_1$ regularization. We provide a theoretical explanation for this observation by analyzing the learning dynamics, revealing that early continuous sparsification is governed by an implicit $L_2$ regularization that gradually transitions to an $L_1$ penalty over time. Leveraging this insight, we propose a method to dynamically control the strength of this implicit bias. Through an extension of the mirror flow framework, we establish convergence and optimality guarantees in the context of underdetermined linear regression. Our theoretical findings may be of independent interest, as we demonstrate how to enter the rich regime and show that the implicit bias can be controlled via a time-dependent Bregman potential. To validate these insights, we introduce PILoT, a continuous sparsification approach with novel initialization and dynamic regularization, which consistently outperforms baselines in standard experiments.