Linguistic Generalizations are not Rules: Impacts on Evaluation of LMs

TL;DR

This paper challenges the assumption that natural language is rule-based and argues that language understanding by LMs involves flexible, context-dependent constructions rather than strict symbolic rules, suggesting new evaluation approaches.

Contribution

It proposes a shift from rule-based to construction-based perspectives for evaluating language models, emphasizing the importance of gradient, context, and function in language understanding.

Findings

Natural languages are not strictly rule-based but rely on flexible constructions.

Failures to obey symbolic rules may reflect language's inherent flexibility.

New benchmarks should consider context and usage variability.

Abstract

Linguistic evaluations of how well LMs generalize to produce or understand language often implicitly take for granted that natural languages are generated by symbolic rules. According to this perspective, grammaticality is determined by whether sentences obey such rules. Interpretation is compositionally generated by syntactic rules operating on meaningful words. Semantic parsing maps sentences into formal logic. Failures of LMs to obey strict rules are presumed to reveal that LMs do not produce or understand language like humans. Here we suggest that LMs' failures to obey symbolic rules may be a feature rather than a bug, because natural languages are not based on neatly separable, compositional rules. Rather, new utterances are produced and understood by a combination of flexible, interrelated, and context-dependent constructions. Considering gradient factors such as frequencies, context, and function will help us reimagine new benchmarks and analyses to probe whether and how LMs capture the rich, flexible generalizations that comprise natural languages.