How Reliable are Causal Probing Interventions?

TL;DR

This paper introduces an empirical framework to evaluate the reliability of causal probing methods in foundation models, revealing tradeoffs between completeness and selectivity, and showing nonlinear methods are generally more reliable.

Contribution

It defines key desiderata for causal probing, introduces a systematic evaluation framework, and compares different methods, highlighting the reliability tradeoff and advantages of nonlinear interventions.

Findings

All methods exhibit a tradeoff between completeness and selectivity.

More complete and reliable methods significantly influence LLM behavior.

Nonlinear interventions are generally more reliable than linear ones.

Abstract

Causal probing aims to analyze foundation models by examining how intervening on their representation of various latent properties impacts their outputs. Recent works have cast doubt on the theoretical basis of several leading causal probing methods, but it has been unclear how to systematically evaluate the effectiveness of these methods in practice. To address this, we define two key causal probing desiderata: completeness (how thoroughly the representation of the target property has been transformed) and selectivity (how little non-targeted properties have been impacted). We find that there is an inherent tradeoff between the two, which we define as reliability, their harmonic mean. We introduce an empirical analysis framework to measure and evaluate these quantities, allowing us to make the first direct comparisons between different families of leading causal probing methods (e.g., linear vs. nonlinear, or concept removal vs. counterfactual interventions). We find that: (1) all methods show a clear tradeoff between completeness and selectivity; (2) more complete and reliable methods have a greater impact on LLM behavior; and (3) nonlinear interventions are almost always more reliable than linear interventions. Our project webpage is available at: https://ahdavies6.github.io/causal_probing_reliability/