Grokking Modular Polynomials

TL;DR

This paper extends analytical solutions for neural networks to learn modular multiplication and polynomials, demonstrating how trained networks can generalize on these tasks and proposing a classification of learnability.

Contribution

It introduces extended analytical solutions for modular multiplication and polynomials, and shows how neural networks can learn and generalize these solutions.

Findings

Networks trained on modular tasks learn solutions similar to analytical ones.

Constructed networks generalize on arbitrary modular polynomials.

Experimental evidence supports classification of learnable vs non-learnable polynomials.

Abstract

Neural networks readily learn a subset of the modular arithmetic tasks, while failing to generalize on the rest. This limitation remains unmoved by the choice of architecture and training strategies. On the other hand, an analytical solution for the weights of Multi-layer Perceptron (MLP) networks that generalize on the modular addition task is known in the literature. In this work, we (i) extend the class of analytical solutions to include modular multiplication as well as modular addition with many terms. Additionally, we show that real networks trained on these datasets learn similar solutions upon generalization (grokking). (ii) We combine these "expert" solutions to construct networks that generalize on arbitrary modular polynomials. (iii) We hypothesize a classification of modular polynomials into learnable and non-learnable via neural networks training; and provide experimental evidence supporting our claims.