APP: Anytime Progressive Pruning
Diganta Misra, Bharat Runwal, Tianlong Chen, Zhangyang Wang, Irina, Rish

TL;DR
This paper introduces Anytime Progressive Pruning (APP), a novel method for training sparse neural networks in online learning settings, achieving significant accuracy improvements and model size reductions across various architectures and datasets.
Contribution
The paper presents a new progressive pruning technique tailored for online learning, outperforming existing dense and baseline models in accuracy and efficiency.
Findings
APP improves accuracy by approximately 7%
Reduces generalization gap by approximately 22%
Achieves comparable performance with only one-third of the dense model size
Abstract
With the latest advances in deep learning, there has been a lot of focus on the online learning paradigm due to its relevance in practical settings. Although many methods have been investigated for optimal learning settings in scenarios where the data stream is continuous over time, sparse networks training in such settings have often been overlooked. In this paper, we explore the problem of training a neural network with a target sparsity in a particular case of online learning: the anytime learning at macroscale paradigm (ALMA). We propose a novel way of progressive pruning, referred to as \textit{Anytime Progressive Pruning} (APP); the proposed approach significantly outperforms the baseline dense and Anytime OSP models across multiple architectures and datasets under short, moderate, and long-sequence training. Our method, for example, shows an improvement in accuracy of $\approx…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Machine Learning and Data Classification · Data Stream Mining Techniques
