Testing Transformer Learnability on the Arithmetic Sequence of Rooted Trees

TL;DR

This paper investigates whether a transformer model can learn the structured sequence of rooted trees generated by prime factorization of natural numbers, revealing partial understanding of the underlying arithmetic grammar.

Contribution

It demonstrates that a GPT-2 model trained on a large sequence of rooted trees can partially learn the internal arithmetic structure and regularities.

Findings

Model captures non-trivial regularities in the sequence

Transformer exhibits partial understanding of arithmetic grammar

Learnability extends beyond empirical data to arithmetic structures

Abstract

We study whether a Large Language Model can learn the deterministic sequence of trees generated by the iterated prime factorization of the natural numbers. Each integer is mapped into a rooted planar tree and the resulting sequence $ \mathbb{N}\mathcal{T}$ defines an arithmetic text with measurable statistical structure. A transformer network (the GPT-2 architecture) is trained from scratch on the first $10^{11}$ elements to subsequently test its predictive ability under next-word and masked-word prediction tasks. Our results show that the model partially learns the internal grammar of $\mathbb{N}\mathcal{T}$, capturing non-trivial regularities and correlations. This suggests that learnability may extend beyond empirical data to the very structure of arithmetic.