Relational Linear Properties in Language Models: An Empirical Investigation

TL;DR

This paper introduces an efficient probing method to empirically test relational linearity in language model representations, revealing layer-wise patterns and sensitivity to phrasing variations.

Contribution

It proposes a novel, more efficient method for testing relational linearity, advancing understanding of how language models encode relational information.

Findings

Relational linearity varies across models and layers.

Layer-wise patterns align with linguistic information distribution.

Relational linearity is affected by paraphrasing of relations.

Abstract

Linear properties are ubiquitous in the representations of language models; however, testing them experimentally remains a challenging task. This work focuses on relational linearity: the hypothesis that, for a fixed relation (e.g., "plays"), the unembedding of an object (e.g., "trumpet") can be predicted from the embedding of its subject (e.g.,"Miles Davis") by a linear map. We present an experimental method to test the formulation of relational linearity by Marconato et al. (2025). Specifically, we introduce a probing method, based on Kullback-Leibler divergence, to evaluate this property and examine its variation across layers and paraphrased relational queries. It is also more efficient than previous work; for example, it avoids the crude Jacobian approximations used in Linear Relational Embeddings by Hernandez et al. (2024). Our findings across four datasets show that relational linearity varies across models, exhibits layer-wise patterns consistent with prior observations about linguistic information in model representations, and is differently affected by changes in how the relation is phrased.