Isolated Causal Effects of Natural Language

TL;DR

This paper introduces a formal framework for estimating the isolated causal effects of language interventions on reader perceptions, addressing challenges of non-focal language approximation and providing measures for bias and sensitivity analysis.

Contribution

It presents a novel estimation framework for isolated causal effects of language, including methods to evaluate approximation quality and bias, validated on real-world and semi-synthetic data.

Findings

Poor non-focal language approximation causes bias in effect estimates

Proposed measures effectively evaluate bias and sensitivity

Framework successfully recovers isolated effects in experiments

Abstract

As language technologies become widespread, it is important to understand how changes in language affect reader perceptions and behaviors. These relationships may be formalized as the isolated causal effect of some focal language-encoded intervention (e.g., factual inaccuracies) on an external outcome (e.g., readers' beliefs). In this paper, we introduce a formal estimation framework for isolated causal effects of language. We show that a core challenge of estimating isolated effects is the need to approximate all non-focal language outside of the intervention. Drawing on the principle of omitted variable bias, we provide measures for evaluating the quality of both non-focal language approximations and isolated effect estimates themselves. We find that poor approximation of non-focal language can lead to bias in the corresponding isolated effect estimates due to omission of relevant variables, and we show how to assess the sensitivity of effect estimates to such bias along the two key axes of fidelity and overlap. In experiments on semi-synthetic and real-world data, we validate the ability of our framework to correctly recover isolated effects and demonstrate the utility of our proposed measures.