# Shared sensitivity to data distribution during learning in humans and transformer networks

## TL;DR

Humans and transformer networks show similar learning patterns when exposed to different data distributions, but only humans benefit from diverse training examples early on.

## Contribution

The study reveals shared sensitivity to data distribution in humans and transformers, with distinct benefits for humans from diverse early training.

## Key findings

- Redundancy and diversity in training data affect in-weights and in-context learning similarly in humans and transformers.
- A balanced mix of redundancy and diversity allows both learning strategies to be used together.
- Humans benefit from diverse examples early in training, while transformers do not.

## Abstract

Do humans learn like transformers? We trained both humans (n = 530) and transformer networks on a rule learning task where they had to respond to a query in a sequence. At test, we measured ‘in-context’ learning (generalize the rule to novel queries) and ‘in-weights’ learning (recall past experiences from memory). Manipulating the diversity and redundancy of examples in the training distribution, we found that humans and transformer networks respond in very similar ways. In both types of learner, redundancy and diversity trade off in driving in-weights and in-context learning, respectively, whereas a composite distribution with a balanced mix of redundancy and diversity allows the two strategies to be used in tandem. However, we also found that while humans benefit from dynamic training schedules that emphasize diverse examples early, transformers do not. So, while the same data-distributional properties promote learning in humans and transformer networks, only people benefit from curricula.

Pesnot Lerousseau and Summerfield compare humans to transformer neural networks in a learning task designed to distinguish ‘in-weights’ and ‘in-context’ learning processes.