Verb Conjugation in Transformers Is Determined by Linear Encodings of Subject Number

TL;DR

This paper demonstrates that BERT encodes subject number linearly for verb conjugation, and this encoding can be manipulated to affect conjugation accuracy, revealing interpretable internal representations.

Contribution

It shows that BERT's verb conjugation depends on a linear encoding of subject number, identified through causal intervention analysis, clarifying the interpretability of linguistic features in transformers.

Findings

Linear encoding of subject number in BERT influences verb conjugation.

Subject number encoding is located at different layers depending on position.

Manipulating the encoding affects conjugation accuracy predictably.

Abstract

Deep architectures such as Transformers are sometimes criticized for having uninterpretable "black-box" representations. We use causal intervention analysis to show that, in fact, some linguistic features are represented in a linear, interpretable format. Specifically, we show that BERT's ability to conjugate verbs relies on a linear encoding of subject number that can be manipulated with predictable effects on conjugation accuracy. This encoding is found in the subject position at the first layer and the verb position at the last layer, but distributed across positions at middle layers, particularly when there are multiple cues to subject number.