Tolerance Principle and Small Language Model Learning

TL;DR

This study investigates whether a small transformer-based language model, BabyBERTa, learns grammatical rules in line with the Tolerance Principle, revealing differences between machine learning and human language acquisition.

Contribution

It provides an empirical test of the Tolerance Principle on a small language model, highlighting discrepancies with human learning patterns.

Findings

BabyBERTa's learning does not align with the Tolerance Principle.

The model's ability to generalize rules depends on training data characteristics.

Results suggest differences between machine and human language learning mechanisms.

Abstract

Modern language models like GPT-3, BERT, and LLaMA require massive training data, yet with sufficient training they reliably learn to distinguish grammatical from ungrammatical sentences. Children aged as young as 14 months already have the capacity to learn abstract grammar rules from very few exemplars, even in the presence of non-rule-following exceptions. Yang's (2016) Tolerance Principle defines a precise threshold for how many exceptions a rule can tolerate and still be learnable. The present study explored the minimal amount and quality of training data necessary for rules to be generalized by a transformer-based language model to test the predictions of the Tolerance Principle. We trained BabyBERTa (Huebner et al. 2021), a transformer model optimized for small datasets, on artificial grammars. The training sets varied in size, number of unique sentence types, and proportion of rule-following versus exception exemplars. We found that, unlike human infants, BabyBERTa's learning dynamics do not align with the Tolerance Principle.