Uncovering a Winning Lottery Ticket with Continuously Relaxed Bernoulli Gates

TL;DR

This paper introduces a fully differentiable method using relaxed Bernoulli gates to efficiently discover sparse subnetworks in neural networks, significantly reducing memory and computation costs while maintaining accuracy.

Contribution

It presents the first fully differentiable approach for Strong Lottery Ticket discovery that avoids non-differentiable estimators and iterative pruning, enabling scalable network sparsification.

Findings

Achieves up to 90% sparsity with minimal accuracy loss

Nearly doubles the sparsity compared to edge-popup at similar accuracy

Works across various architectures including CNNs and Vision Transformers

Abstract

Over-parameterized neural networks incur prohibitive memory and computational costs for resource-constrained deployment. The Strong Lottery Ticket (SLT) hypothesis suggests that randomly initialized networks contain sparse subnetworks achieving competitive accuracy without weight training. Existing SLT methods, notably edge-popup, rely on non-differentiable score-based selection, limiting optimization efficiency and scalability. We propose using continuously relaxed Bernoulli gates to discover SLTs through fully differentiable, end-to-end optimization - training only gating parameters while keeping all network weights frozen at their initialized values. Continuous relaxation enables direct gradient-based optimization of an $\ell_0$-regularization objective, eliminating the need for non-differentiable gradient estimators or iterative pruning cycles. To our knowledge, this is the first fully differentiable approach for SLT discovery that avoids straight-through estimator approximations. Experiments across fully connected networks, CNNs (ResNet, Wide-ResNet), and Vision Transformers (ViT, Swin-T) demonstrate up to 90% sparsity with minimal accuracy loss - nearly double the sparsity achieved by edge-popup at comparable accuracy - establishing a scalable framework for pre-training network sparsification.