Dynamic Dropout: Leveraging Conway's Game of Life for Neural Networks Regularization

TL;DR

This paper introduces a novel regularization method for neural networks that replaces traditional dropout with Conway's Game of Life, enabling dynamic, pattern-driven deactivation of units to improve generalization.

Contribution

The paper proposes using Conway's Game of Life for dynamic neural network regularization, offering an interpretable and adaptable alternative to static dropout.

Findings

Achieves comparable performance to dropout on CIFAR-10

Visualizes evolving spatial patterns in network units

Enhances deeper neural architectures' performance

Abstract

Regularization techniques play a crucial role in preventing overfitting and improving the generalization performance of neural networks. Dropout, a widely used regularization technique, randomly deactivates units during training to introduce redundancy and prevent co-adaptation among neurons. Despite its effectiveness, dropout has limitations, such as its static nature and lack of interpretability. In this paper, we propose a novel approach to regularization by substituting dropout with Conway's Game of Life (GoL), a cellular automata with simple rules that govern the evolution of a grid of cells. We introduce dynamic unit deactivation during training by representing neural network units as cells in a GoL grid and applying the game's rules to deactivate units. This approach allows for the emergence of spatial patterns that adapt to the training data, potentially enhancing the network's ability to generalize. We demonstrate the effectiveness of our approach on the CIFAR-10 dataset, showing that dynamic unit deactivation using GoL achieves comparable performance to traditional dropout techniques while offering insights into the network's behavior through the visualization of evolving patterns. Furthermore, our discussion highlights the applicability of our proposal in deeper architectures, demonstrating how it enhances the performance of different dropout techniques.