Meta predictive learning model of languages in neural circuits

TL;DR

This paper introduces a meta predictive learning model based on the predictive coding framework, mimicking brain-like language processing, validated on digit classification and language tasks, and showing emergent behaviors similar to large language models.

Contribution

It proposes a novel mean-field learning model within the predictive coding framework that trains distributions of synaptic weights, bridging brain computation and language modeling.

Findings

Connections become deterministic after learning

Output connections exhibit higher variability

Model performance improves with more data

Abstract

Large language models based on self-attention mechanisms have achieved astonishing performances not only in natural language itself, but also in a variety of tasks of different nature. However, regarding processing language, our human brain may not operate using the same principle. Then, a debate is established on the connection between brain computation and artificial self-supervision adopted in large language models. One of most influential hypothesis in brain computation is the predictive coding framework, which proposes to minimize the prediction error by local learning. However, the role of predictive coding and the associated credit assignment in language processing remains unknown. Here, we propose a mean-field learning model within the predictive coding framework, assuming that the synaptic weight of each connection follows a spike and slab distribution, and only the distribution, rather than specific weights, is trained. This meta predictive learning is successfully validated on classifying handwritten digits where pixels are input to the network in sequence, and moreover on the toy and real language corpus. Our model reveals that most of the connections become deterministic after learning, while the output connections have a higher level of variability. The performance of the resulting network ensemble changes continuously with data load, further improving with more training data, in analogy with the emergent behavior of large language models. Therefore, our model provides a starting point to investigate the connection among brain computation, next-token prediction and general intelligence.