Causal Interventions Reveal Shared Structure Across English Filler-Gap Constructions

TL;DR

This paper uses causal interpretability methods on language models to uncover shared underlying structures in English filler-gap constructions, revealing insights that can inform and improve linguistic theories.

Contribution

It introduces a novel application of causal interventions to analyze language models' understanding of filler-gap dependencies, uncovering shared structures and overlooked factors.

Findings

Language models converge on similar analyses of filler-gap constructions

Identified factors like frequency, filler type, and context influence model analyses

Results suggest mechanistic analyses can advance linguistic theory

Abstract

Language Models (LMs) have emerged as powerful sources of evidence for linguists seeking to develop theories of syntax. In this paper, we argue that causal interpretability methods, applied to LMs, can greatly enhance the value of such evidence by helping us characterize the abstract mechanisms that LMs learn to use. Our empirical focus is a set of English filler-gap dependency constructions (e.g., questions, relative clauses). Linguistic theories largely agree that these constructions share many properties. Using experiments based in Distributed Interchange Interventions, we show that LMs converge on similar abstract analyses of these constructions. These analyses also reveal previously overlooked factors -- relating to frequency, filler type, and surrounding context -- that could motivate changes to standard linguistic theory. Overall, these results suggest that mechanistic, internal analyses of LMs can push linguistic theory forward.