Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training

TL;DR

This paper introduces a causal intervention framework to improve subword-based language models' ability to handle character-level tasks by learning interpretable character representations, enhancing robustness and performance on complex form-meaning tasks.

Contribution

It develops a novel causal intervention method for character representation learning within subword models and introduces new character-level tasks to evaluate their capabilities.

Findings

Outperforms standard models on complex character tasks

Improves robustness to unseen token sequences

Produces human-interpretable character representations

Abstract

Language tasks involving character-level manipulations (e.g., spelling corrections, arithmetic operations, word games) are challenging for models operating on subword units. To address this, we develop a causal intervention framework to learn robust and interpretable character representations inside subword-based language models. Our method treats each character as a typed variable in a causal model and learns such causal structures by adapting the interchange intervention training method of Geiger et al. (2021). We additionally introduce a suite of character-level tasks that systematically vary in their dependence on meaning and sequence-level context. While character-level models still perform best on purely form-based tasks like string reversal, our method outperforms character-level models on more complex tasks that blend form, meaning, and context, such as spelling correction in context and word search games. Compared with standard subword-based models, our approach also significantly improves robustness on unseen token sequences and leads to human-interpretable internal representations of characters.