Fitting Multilinear Polynomials for Logic Gate Networks

TL;DR

This paper introduces a novel approach for learning logic gate networks by leveraging multilinear polynomial representations, addressing gradient vanishing issues in existing methods, and demonstrating improved performance across multiple datasets.

Contribution

The authors propose a 4-dimensional polynomial space method that outperforms traditional Soft-Mix in logic gate network training, especially at greater depths.

Findings

CovJac method maintains performance at depth, unlike Soft-Mix.

Working in 4D reduces parameters from 16 to 4 per neuron.

CovJac outperforms Soft-Mix on all seven datasets.

Abstract

We study learnable logic gate networks that stack layers of 2-input Boolean gates to build combinational circuits. Every 2-input gate has a unique multilinear polynomial with 4 coefficients, so the 16 Boolean gates form a codebook of prototypes in a 4-dimensional space, reducing training to a vector-quantization problem. The baseline method, Soft-Mix, learns a 16-dimensional softmax over gate identities, but the codebook has rank~4: 11 of 15 simplex directions carry nullspace gradient, and at uniform initialization the backward signal vanishes exactly. We prove that no affine product reparameterization fixes the resulting interaction-coefficient starvation under STE, and show that the covariance Jacobian of soft-VQ selection bypasses it by coupling the starved coefficient to the always-active constant channel. Working in the 4-dimensional polynomial space reduces each neuron from 16 to 4 parameters. On seven datasets, at least one 4-parameter method matches or exceeds Soft-Mix on every dataset; the CovJac advantage over STE grows monotonically with interaction demand across all seven datasets. At depth, Soft-Mix collapses ($-37.3$pp on CIFAR-10 at 12 layers) while CovJac holds ($-0.5$pp on CIFAR-10, stable on MNIST).