Optimizing Learning Rate Schedules for Iterative Pruning of Deep Neural Networks

TL;DR

This paper introduces SILO, a theoretically motivated learning rate schedule for neural network pruning that dynamically adjusts the LR to improve generalization, achieving results comparable to exhaustive search methods.

Contribution

The paper provides a theoretical justification for LR schedules in pruning and proposes SILO, a new schedule that adaptively adjusts LR for better performance and generalization.

Findings

SILO improves accuracy by 2-4% on ImageNet and CIFAR datasets.

SILO matches Oracle performance with lower complexity.

Theoretical analysis supports the effectiveness of SILO's LR adjustment.

Abstract

The importance of learning rate (LR) schedules on network pruning has been observed in a few recent works. As an example, Frankle and Carbin (2019) highlighted that winning tickets (i.e., accuracy preserving subnetworks) can not be found without applying a LR warmup schedule and Renda, Frankle and Carbin (2020) demonstrated that rewinding the LR to its initial state at the end of each pruning cycle improves performance. In this paper, we go one step further by first providing a theoretical justification for the surprising effect of LR schedules. Next, we propose a LR schedule for network pruning called SILO, which stands for S-shaped Improved Learning rate Optimization. The advantages of SILO over existing state-of-the-art (SOTA) LR schedules are two-fold: (i) SILO has a strong theoretical motivation and dynamically adjusts the LR during pruning to improve generalization. Specifically, SILO increases the LR upper bound (max_lr) in an S-shape. This leads to an improvement of 2% - 4% in extensive experiments with various types of networks (e.g., Vision Transformers, ResNet) on popular datasets such as ImageNet, CIFAR-10/100. (ii) In addition to the strong theoretical motivation, SILO is empirically optimal in the sense of matching an Oracle, which exhaustively searches for the optimal value of max_lr via grid search. We find that SILO is able to precisely adjust the value of max_lr to be within the Oracle optimized interval, resulting in performance competitive with the Oracle with significantly lower complexity.