A Group Theoretic Analysis of the Symmetries Underlying Base Addition and Their Learnability by Neural Networks

TL;DR

This paper uses group theory to analyze the symmetries in base addition, focusing on carry functions, and investigates how neural networks learn these symmetries, revealing that carry structure significantly influences learnability and generalization.

Contribution

It introduces a group theoretic framework for analyzing carry functions in base addition and examines neural network learnability of these symmetries based on carry structure.

Findings

Neural networks can achieve radical generalization with proper input format and carry function.

Carry function structure significantly affects neural network learnability.

Different carry functions lead to varying rates of learning in neural networks.

Abstract

A major challenge in the use of neural networks both for modeling human cognitive function and for artificial intelligence is the design of systems with the capacity to efficiently learn functions that support radical generalization. At the roots of this is the capacity to discover and implement symmetry functions. In this paper, we investigate a paradigmatic example of radical generalization through the use of symmetry: base addition. We present a group theoretic analysis of base addition, a fundamental and defining characteristic of which is the carry function -- the transfer of the remainder, when a sum exceeds the base modulus, to the next significant place. Our analysis exposes a range of alternative carry functions for a given base, and we introduce quantitative measures to characterize these. We then exploit differences in carry functions to probe the inductive biases of neural networks in symmetry learning, by training neural networks to carry out base addition using different carries, and comparing efficacy and rate of learning as a function of their structure. We find that even simple neural networks can achieve radical generalization with the right input format and carry function, and that learnability is closely correlated with carry function structure. We then discuss the relevance this has for cognitive science and machine learning.