Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaning

TL;DR

This paper investigates whether human-scale language models can generalize knowledge of rare grammatical constructions, specifically LET-ALONE, in both form and meaning, revealing they understand form but not meaning, unlike humans.

Contribution

It introduces a synthetic benchmark to evaluate language models' understanding of rare constructions' form and meaning, highlighting an asymmetry in their generalization abilities.

Findings

Models recognize form but not meaning of LET-ALONE

Models generalize form from frequent to rare constructions

Humans understand both form and meaning of rare constructions

Abstract

Humans have a remarkable ability to acquire and understand grammatical phenomena that are seen rarely, if ever, during childhood. Recent evidence suggests that language models with human-scale pretraining data may possess a similar ability by generalizing from frequent to rare constructions. However, it remains an open question how widespread this generalization ability is, and to what extent this knowledge extends to meanings of rare constructions, as opposed to just their forms. We fill this gap by testing human-scale transformer language models on their knowledge of both the form and meaning of the (rare and quirky) English LET-ALONE construction. To evaluate our LMs we construct a bespoke synthetic benchmark that targets syntactic and semantic properties of the construction. We find that human-scale LMs are sensitive to form, even when related constructions are filtered from the dataset. However, human-scale LMs do not make correct generalizations about LET-ALONE's meaning. These results point to an asymmetry in the current architectures' sample efficiency between language form and meaning, something which is not present in human language learners.