Towards Scalable Lottery Ticket Networks using Genetic Algorithms

TL;DR

This paper proposes using genetic algorithms to identify lottery ticket subnetworks in neural networks, achieving high accuracy and sparsity without training, thus enabling scalable and efficient deep learning models.

Contribution

It introduces a novel approach employing genetic algorithms to find lottery ticket subnetworks, outperforming existing methods in accuracy and sparsity without gradient-based training.

Findings

Achieves better accuracy and sparsity than state-of-the-art methods.

Does not require gradient information for subnetwork identification.

Effective for binary and multi-class classification tasks.

Abstract

Building modern deep learning systems that are not just effective but also efficient requires rethinking established paradigms for model training and neural architecture design. Instead of adapting highly overparameterized networks and subsequently applying model compression techniques to reduce resource consumption, a new class of high-performing networks skips the need for expensive parameter updates, while requiring only a fraction of parameters, making them highly scalable. The Strong Lottery Ticket Hypothesis posits that within randomly initialized, sufficiently overparameterized neural networks, there exist subnetworks that can match the accuracy of the trained original model-without any training. This work explores the usage of genetic algorithms for identifying these strong lottery ticket subnetworks. We find that for instances of binary and multi-class classification tasks, our approach achieves better accuracies and sparsity levels than the current state-of-the-art without requiring any gradient information. In addition, we provide justification for the need for appropriate evaluation metrics when scaling to more complex network architectures and learning tasks.