Mind the Gap: Removing the Discretization Gap in Differentiable Logic Gate Networks

TL;DR

This paper introduces a method using Gumbel noise and a straight-through estimator to significantly speed up training, reduce the discretization gap, and improve the efficiency of logic gate networks for image classification.

Contribution

It proposes a novel training technique that accelerates LGN training, minimizes the discretization gap, and enhances neuron utilization, supported by theoretical analysis.

Findings

Training speed increased by 4.5 times

Discretization gap reduced by 98%

Unused gates eliminated completely

Abstract

Modern neural networks demonstrate state-of-the-art performance on numerous existing benchmarks; however, their high computational requirements and energy consumption prompt researchers to seek more efficient solutions for real-world deployment. Logic gate networks (LGNs) learns a large network of logic gates for efficient image classification. However, learning a network that can solve a simple problem like CIFAR-10 can take days to weeks to train. Even then, almost half of the network remains unused, causing a discretization gap. This discretization gap hinders real-world deployment of LGNs, as the performance drop between training and inference negatively impacts accuracy. We inject Gumbel noise with a straight-through estimator during training to significantly speed up training, improve neuron utilization, and decrease the discretization gap. We theoretically show that this results from implicit Hessian regularization, which improves the convergence properties of LGNs. We train networks $4.5 \times$ faster in wall-clock time, reduce the discretization gap by $98\%$, and reduce the number of unused gates by $100\%$.