One Language, Two Scripts: Probing Script-Invariance in LLM Concept Representations

TL;DR

This study investigates whether Sparse Autoencoders in large language models encode abstract meaning rather than script-specific features, using Serbian digraphia as a controlled testbed to demonstrate the models' script-invariant semantic representations.

Contribution

The paper introduces Serbian digraphia as a novel evaluation paradigm and provides evidence that SAE features in LLMs prioritize meaning over orthography, especially as model scale increases.

Findings

SAE features activate similarly across different Serbian scripts.

Changing script causes less divergence than paraphrasing within the same script.

Model scale enhances script-invariance in representations.

Abstract

Do the features learned by Sparse Autoencoders (SAEs) represent abstract meaning, or are they tied to how text is written? We investigate this question using Serbian digraphia as a controlled testbed: Serbian is written interchangeably in Latin and Cyrillic scripts with a near-perfect character mapping between them, enabling us to vary orthography while holding meaning exactly constant. Crucially, these scripts are tokenized completely differently, sharing no tokens whatsoever. Analyzing SAE feature activations across the Gemma model family (270M-27B parameters), we find that identical sentences in different Serbian scripts activate highly overlapping features, far exceeding random baselines. Strikingly, changing script causes less representational divergence than paraphrasing within the same script, suggesting SAE features prioritize meaning over orthographic form. Cross-script cross-paraphrase comparisons provide evidence against memorization, as these combinations rarely co-occur in training data yet still exhibit substantial feature overlap. This script invariance strengthens with model scale. Taken together, our findings suggest that SAE features can capture semantics at a level of abstraction above surface tokenization, and we propose Serbian digraphia as a general evaluation paradigm for probing the abstractness of learned representations.