A Geometric Notion of Causal Probing

TL;DR

This paper introduces an intrinsic, geometric, and information-theoretic approach to identify and manipulate concept subspaces in language models, demonstrating effective concept erasure and controlled generation without auxiliary classifiers.

Contribution

It proposes a set of intrinsic criteria for ideal linear concept subspaces, enabling their identification solely from model distributions, advancing understanding of concept encoding in language models.

Findings

Linear concept erasure effectively removes concept information.

Concept subspaces can be used for precise manipulation of generated words.

Framework accounts for spurious correlations in representation spaces.

Abstract

The linear subspace hypothesis (Bolukbasi et al., 2016) states that, in a language model's representation space, all information about a concept such as verbal number is encoded in a linear subspace. Prior work has relied on auxiliary classification tasks to identify and evaluate candidate subspaces that might give support for this hypothesis. We instead give a set of intrinsic criteria which characterize an ideal linear concept subspace and enable us to identify the subspace using only the language model distribution. Our information-theoretic framework accounts for spuriously correlated features in the representation space (Kumar et al., 2022) by reconciling the statistical notion of concept information and the geometric notion of how concepts are encoded in the representation space. As a byproduct of this analysis, we hypothesize a causal process for how a language model might leverage concepts during generation. Empirically, we find that linear concept erasure is successful in erasing most concept information under our framework for verbal number as well as some complex aspect-level sentiment concepts from a restaurant review dataset. Our causal intervention for controlled generation shows that, for at least one concept across two languages models, the concept subspace can be used to manipulate the concept value of the generated word with precision.