A Universal Vibe? Finding and Controlling Language-Agnostic Informal Register with SAEs

TL;DR

This paper investigates whether multilingual language models understand informal registers like slang as abstract, language-agnostic concepts, revealing a shared core that influences output formality across languages.

Contribution

It introduces a novel dataset and demonstrates that models internalize informal register as a transferable, language-agnostic pragmatic abstraction, not just surface-level features.

Findings

A robust cross-linguistic informal register core exists in model representations.

Activation steering can causally shift output formality across languages.

Shared informal register representations transfer zero-shot to unseen languages.

Abstract

While multilingual language models successfully transfer factual and syntactic knowledge across languages, it remains unclear whether they process culture-specific pragmatic registers, such as slang, as isolated language-specific memorizations or as unified, abstract concepts. We study this by probing the internal representations of Gemma-2-9B-IT using Sparse Autoencoders (SAEs) across three typologically diverse source languages: English, Hebrew, and Russian. To definitively isolate pragmatic register processing from trivial lexical sensitivity, we introduce a novel dataset in which every target term is polysemous, appearing in both literal and informal contexts. We find that while much of the informal-register signal is distributed across language-specific features, a small but highly robust cross-linguistic core consistently emerges. This shared core forms a geometrically coherent ``informal register subspace'' that sharpens in the model's deeper layers. Crucially, these shared representations are not merely correlational: activation steering with these features causally shifts output formality across all source languages and transfers zero-shot to six unseen languages spanning diverse language families and scripts. Together, these results provide the first mechanistic evidence that multilingual LLMs internalize informal register not just as surface-level heuristics, but as a portable, language-agnostic pragmatic abstraction.