Multilingual Embedding Probes Fail to Generalize Across Learner Corpora

TL;DR

This study evaluates whether multilingual embedding models encode a universal proficiency representation by probing their hidden states across multiple languages and corpora, revealing limited generalization.

Contribution

It demonstrates that current multilingual embeddings fail to produce transferable proficiency representations across different learner corpora.

Findings

Probes perform well within the same corpus but fail cross-corpus.

Probes tend to predict uniform labels out-of-distribution, indicating corpus-specific encoding.

Multilingual embeddings do not straightforwardly encode a language-general proficiency dimension.

Abstract

Do multilingual embedding models encode a language-general representation of proficiency? We investigate this by training linear and non-linear probes on hidden-state activations from Qwen3-Embedding (0.6B, 4B, 8B) to predict CEFR proficiency levels from learner texts across nine corpora and seven languages. We compare five probing architectures against a baseline trained on surface-level text features. Under in-distribution evaluation, probes achieve strong performance ($QWK\approx0.7$), substantially outperforming the surface baseline, with middle layers consistently yielding the best predictions. However, in cross-corpus evaluation performance collapses across all probe types and model sizes. Residual analysis reveals that out-of-distribution probes converge towards predicting uniformly distributed labels, indicating that the learned mappings capture corpus-specific distributional properties (topic, language, task type, rating methodology) rather than an abstract, transferable proficiency dimension. These results suggest that current multilingual embeddings do not straightforwardly encode language-general proficiency, with implications for representation-based approaches to proficiency-adaptive language technology.