BINGO: A Novel Pruning Mechanism to Reduce the Size of Neural Networks

TL;DR

BINGO introduces a one-pass, significance-based pruning method during training that reduces neural network size efficiently without sacrificing accuracy, addressing high computational costs of traditional pruning techniques.

Contribution

BINGO presents a novel pruning mechanism that assesses weight significance during training for single-shot pruning, reducing computational and environmental costs.

Findings

Maintains accuracy with significant model size reduction

Less computationally intensive than iterative pruning methods

Enables scalable AI development without increasing model size

Abstract

Over the past decade, the use of machine learning has increased exponentially. Models are far more complex than ever before, growing to gargantuan sizes and housing millions of weights. Unfortunately, the fact that large models have become the state of the art means that it often costs millions of dollars to train and operate them. These expenses not only hurt companies but also bar non-wealthy individuals from contributing to new developments and force consumers to pay greater prices for AI. Current methods used to prune models, such as iterative magnitude pruning, have shown great accuracy but require an iterative training sequence that is incredibly computationally and environmentally taxing. To solve this problem, BINGO is introduced. BINGO, during the training pass, studies specific subsets of a neural network one at a time to gauge how significant of a role each weight plays in contributing to a network's accuracy. By the time training is done, BINGO generates a significance score for each weight, allowing for insignificant weights to be pruned in one shot. BINGO provides an accuracy-preserving pruning technique that is less computationally intensive than current methods, allowing for a world where AI growth does not have to mean model growth, as well.