Learning to Add, Multiply, and Execute Algorithmic Instructions Exactly with Neural Networks

TL;DR

This paper demonstrates that neural networks, specifically in the NTK regime, can learn to execute fundamental binary operations and algorithms exactly, with minimal training data, by leveraging structured data and correlation control.

Contribution

It introduces a method using NTK theory to train neural networks to perform exact binary algorithmic tasks with high probability, extending to Turing-complete functions.

Findings

Neural networks can learn to perform binary permutations, addition, and multiplication exactly.

A logarithmic amount of training data suffices for the networks to generalize.

The approach applies to Turing-complete functions like SBN instructions.

Abstract

Neural networks are known for their ability to approximate smooth functions, yet they fail to generalize perfectly to unseen inputs when trained on discrete operations. Such operations lie at the heart of algorithmic tasks such as arithmetic, which is often used as a test bed for algorithmic execution in neural networks. In this work, we ask: can neural networks learn to execute binary-encoded algorithmic instructions exactly? We use the Neural Tangent Kernel (NTK) framework to study the training dynamics of two-layer fully connected networks in the infinite-width limit and show how a sufficiently large ensemble of such models can be trained to execute exactly, with high probability, four fundamental tasks: binary permutations, binary addition, binary multiplication, and Subtract and Branch if Negative (SBN) instructions. Since SBN is Turing-complete, our framework extends to computable functions. We show how this can be efficiently achieved using only logarithmically many training data. Our approach relies on two techniques: structuring the training data to isolate bit-level rules, and controlling correlations in the NTK regime to align model predictions with the target algorithmic executions.