What Makes Language Models Good-enough?

TL;DR

This paper investigates how architectural features like layers and attention heads in Transformers influence their ability to perform good-enough language processing, inspired by human psycholinguistic findings.

Contribution

It introduces the GELP dataset and demonstrates that shallower models with fewer attention heads can still achieve good-enough language understanding.

Findings

Shallower models perform comparably to deeper ones on GELP.

Fewer self-attention heads do not significantly impair performance.

The GELP dataset effectively evaluates good-enough language processing.

Abstract

Psycholinguistic research suggests that humans may build a representation of linguistic input that is 'good-enough' for the task at hand. This study examines what architectural features make language models learn human-like good-enough language processing. We focus on the number of layers and self-attention heads in Transformers. We create a good-enough language processing (GELP) evaluation dataset (7,680 examples), which is designed to test the effects of two plausibility types, eight construction types, and three degrees of memory cost on language processing. To annotate GELP, we first conduct a crowdsourcing experiment whose design follows prior psycholinguistic studies. Our model evaluation against the annotated GELP then reveals that the full model as well as models with fewer layers and/or self-attention heads exhibit a good-enough performance. This result suggests that models with shallower depth and fewer heads can learn good-enough language processing.