Learning Digital Circuits: A Journey Through Weight Invariant Self-Pruning Neural Networks

TL;DR

This paper introduces a method for designing neural networks with binary weights that act as digital circuits, achieving high pruning rates and digital-like neuron behavior while maintaining performance on vision tasks.

Contribution

The authors propose a weight-invariant self-pruning neural network approach using binarized weights, revealing digital circuit-like neuron behavior and reducing weights by over 99%.

Findings

Achieves similar performance to standard networks with over 99% weight pruning.

Neurons behave like NOR gates, functioning as digital circuits.

Networks can perform tasks with constant weights without tuning.

Abstract

Recently, in the paper "Weight Agnostic Neural Networks" Gaier & Ha utilized architecture search to find networks where the topology completely encodes the knowledge. However, architecture search in topology space is expensive. We use the existing framework of binarized networks to find performant topologies by constraining the weights to be either, zero or one. We show that such topologies achieve performance similar to standard networks while pruning more than 99% weights. We further demonstrate that these topologies can perform tasks using constant weights without any explicit tuning. Finally, we discover that in our setup each neuron acts like a NOR gate, virtually learning a digital circuit. We demonstrate the efficacy of our approach on computer vision datasets.