LOFT: Finding Lottery Tickets through Filter-wise Training
Qihan Wang, Chen Dun, Fangshuo Liao, Chris Jermaine, Anastasios, Kyrillidis

TL;DR
This paper introduces LOFT, a filter-wise training method for efficiently identifying lottery tickets in CNNs, reducing pretraining costs while maintaining accuracy.
Contribution
The paper proposes a novel filter distance metric and a distributed pretraining algorithm, LOFT, that efficiently finds lottery tickets without extensive pretraining.
Findings
LOFT preserves and finds high-quality lottery tickets.
LOFT achieves significant computation and communication savings.
LOFT maintains or improves accuracy compared to other pretraining methods.
Abstract
Recent work on the Lottery Ticket Hypothesis (LTH) shows that there exist ``\textit{winning tickets}'' in large neural networks. These tickets represent ``sparse'' versions of the full model that can be trained independently to achieve comparable accuracy with respect to the full model. However, finding the winning tickets requires one to \emph{pretrain} the large model for at least a number of epochs, which can be a burdensome task, especially when the original neural network gets larger. In this paper, we explore how one can efficiently identify the emergence of such winning tickets, and use this observation to design efficient pretraining algorithms. For clarity of exposition, our focus is on convolutional neural networks (CNNs). To identify good filters, we propose a novel filter distance metric that well-represents the model convergence. As our theory dictates, our filter…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeural Networks and Applications · Stock Market Forecasting Methods · Generative Adversarial Networks and Image Synthesis
