Measure More, Question More: Experimental Studies on Transformer-based Language Models and Complement Coercion

TL;DR

This paper explores how transformer-based language models handle implicit meanings in sentences involving complement coercion, revealing their ability to recover implicit information and highlighting areas for improvement.

Contribution

It provides the first detailed analysis of transformer models' responses to implicit meaning, using complement coercion as a case study, and introduces follow-up experiments to refine understanding.

Findings

Models show surprisal effects at regions with implicit meaning

Follow-up experiments clarify effects are not due to confounds

Results suggest models partially recover implicit information

Abstract

Transformer-based language models have shown strong performance on an array of natural language understanding tasks. However, the question of how these models react to implicit meaning has been largely unexplored. We investigate this using the complement coercion phenomenon, which involves sentences like "The student finished the book about sailing" where the action "reading" is implicit. We compare LMs' surprisal estimates at various critical sentence regions in sentences with and without implicit meaning. Effects associated with recovering implicit meaning were found at a critical region other than where sentences minimally differ. We then use follow-up experiments to factor out potential confounds, revealing different perspectives that offer a richer and more accurate picture.