A framework for analyzing concept representations in neural models

TL;DR

This paper introduces a unified framework to analyze how neural models represent concepts, focusing on containment and disentanglement, and evaluates different estimators and models in text and speech domains.

Contribution

It proposes a novel framework for analyzing concept subspaces in neural models and compares multiple estimators, revealing their impact on concept representation properties.

Findings

Concept subspaces are not always uniquely determined.

Estimator choice affects containment and disentanglement properties.

LEACE performs well but struggles with unseen data.

Abstract

Understanding how neural models represent human-interpretable concepts is challenging. Prior work has explored linear concept subspaces from diverse perspectives, such as probing and concept erasure. We introduce a unified framework to study these subspaces along two axes: \textit{containment}, which tests if a concept is fully represented in a subspace but not outside it, and \textit{disentanglement}, which tests for isolation from other concepts. In experiments on both text and speech models, we first highlight that concept subspaces may not be uniquely determined, and discuss the implications for concept subspace analysis. Then, we compare properties of concept subspaces estimated using five estimators, proposed in different communities. We find that (1) the choice of estimator impacts the containment and disentanglement properties; (2) the state-of-the-art concept erasure method, LEACE, performs well on both testing axes, but still struggles to generalize to unseen data; and (3) in HuBERT speech representations, phone information is both contained and disentangled from speaker information, while speaker information is hard to contain in a compact subspace, despite being disentangled from phones.