Emergence of Phonemic, Syntactic, and Semantic Representations in Artificial Neural Networks

TL;DR

This study demonstrates that artificial neural networks develop phonemic, lexical, and syntactic representations in a sequence during training, mirroring aspects of human language acquisition but requiring significantly more data.

Contribution

It provides a unifying computational framework showing how neural representations of language structures emerge in artificial networks over training.

Findings

Neural activations form structured subspaces representing language features.

Language representations emerge in a sequence during training.

Significant data is needed for these representations to develop.

Abstract

During language acquisition, children successively learn to categorize phonemes, identify words, and combine them with syntax to form new meaning. While the development of this behavior is well characterized, we still lack a unifying computational framework to explain its underlying neural representations. Here, we investigate whether and when phonemic, lexical, and syntactic representations emerge in the activations of artificial neural networks during their training. Our results show that both speech- and text-based models follow a sequence of learning stages: during training, their neural activations successively build subspaces, where the geometry of the neural activations represents phonemic, lexical, and syntactic structure. While this developmental trajectory qualitatively relates to children's, it is quantitatively different: These algorithms indeed require two to four orders of magnitude more data for these neural representations to emerge. Together, these results show conditions under which major stages of language acquisition spontaneously emerge, and hence delineate a promising path to understand the computations underpinning language acquisition.