Diluting Restricted Boltzmann Machines

TL;DR

This study shows that Restricted Boltzmann Machines can retain high performance with significant pruning, but retraining after pruning does not fully recover lost quality, emphasizing early pruning for efficiency.

Contribution

It demonstrates the viability of extreme pruning in RBMs and highlights the importance of early pruning over retraining for maintaining generative performance.

Findings

RBMs can achieve high-quality generation with up to 80% pruning.

Retraining after pruning does not fully recover performance.

Early pruning is more effective than post-training pruning.

Abstract

Recent advances in artificial intelligence have relied heavily on increasingly large neural networks, raising concerns about their computational and environmental costs. This paper investigates whether simpler, sparser networks can maintain strong performance by studying Restricted Boltzmann Machines (RBMs) under extreme pruning conditions. Inspired by the Lottery Ticket Hypothesis, we demonstrate that RBMs can achieve high-quality generative performance even when up to 80% of the connections are pruned before training, confirming that they contain viable sub-networks. However, our experiments reveal crucial limitations: trained networks cannot fully recover lost performance through retraining once additional pruning is applied. We identify a sharp transition above which the generative quality degrades abruptly when pruning disrupts a minimal core of essential connections. Moreover, re-trained networks remain constrained by the parameters originally learned performing worse than networks trained from scratch at equivalent sparsity levels. These results suggest that for sparse networks to work effectively, pruning should be implemented early in training rather than attempted afterwards. Our findings provide practical insights for the development of efficient neural architectures and highlight the persistent influence of initial conditions on network capabilities.