Generics are puzzling. Can language models find the missing piece?

TL;DR

This paper investigates how language models understand generics, revealing their context sensitivity, the expression of weak generalizations, and the reflection of stereotypes, through a new dataset and surprisal-based metric.

Contribution

The study introduces ConGen, a dataset of generic sentences, and a surprisal-based metric to analyze generics' context sensitivity and implicit quantification in language models.

Findings

Generics are more context-sensitive than determiner quantifiers.

Approximately 20% of generics express weak generalizations.

Language models reflect human stereotypes in their responses.

Abstract

Generic sentences express generalisations about the world without explicit quantification. Although generics are central to everyday communication, building a precise semantic framework has proven difficult, in part because speakers use generics to generalise properties with widely different statistical prevalence. In this work, we study the implicit quantification and context-sensitivity of generics by leveraging language models as models of language. We create ConGen, a dataset of 2873 naturally occurring generic and quantified sentences in context, and define p-acceptability, a metric based on surprisal that is sensitive to quantification. Our experiments show generics are more context-sensitive than determiner quantifiers and about 20% of naturally occurring generics we analyze express weak generalisations. We also explore how human biases in stereotypes can be observed in language models.