Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation

TL;DR

This paper investigates whether function vectors from multilingual language models are language-agnostic, demonstrating their transferability across languages and tasks in machine translation, with implications for model generalization.

Contribution

It provides the first systematic analysis of language-agnosticity in function vectors, showing their transferability across languages and model variants in translation tasks.

Findings

Translation FVs from one language improve performance in unseen languages.

Removing FVs degrades translation accuracy, especially in low-resource languages.

FVs transfer from base models to instruction-tuned variants and generalize from word to sentence translation.

Abstract

Function vectors (FVs) are vector representations of tasks extracted from model activations during in-context learning. While prior work has shown that multilingual model representations can be language-agnostic, it remains unclear whether the same holds for function vectors. We study whether FVs exhibit language-agnosticity, using machine translation as a case study. Across three decoder-only multilingual LLMs, we find that translation FVs extracted from a single English$\rightarrow$Target direction transfer to other target languages, consistently improving the rank of correct translation tokens across multiple unseen languages. Ablation results show that removing the FV degrades translation across languages with limited impact on unrelated tasks. We further show that base-model FVs transfer to instruction-tuned variants and partially generalize from word-level to sentence-level translation.