Semantic Pivots Enable Cross-Lingual Transfer in Large Language Models

TL;DR

This paper investigates how large language models acquire cross-lingual abilities by analyzing intermediate layer behaviors, identifying semantic pivots, and reconstructing datasets to enhance multilingual transfer.

Contribution

It introduces a novel Word-Level Cross-Lingual Translation Task and a semantic pivot-aware dataset to improve LLMs' cross-lingual transfer capabilities.

Findings

Identified co-occurrence and semantic pivot behaviors in LLMs

Semantic pivot-based dataset improves cross-lingual performance

Enhanced interpretability of LLMs in multilingual tasks

Abstract

Large language models (LLMs) demonstrate remarkable ability in cross-lingual tasks. Understanding how LLMs acquire this ability is crucial for their interpretability. To quantify the cross-lingual ability of LLMs accurately, we propose a Word-Level Cross-Lingual Translation Task. To find how LLMs learn cross-lingual ability, we trace the outputs of LLMs' intermediate layers in the word translation task. We identify and distinguish two distinct behaviors in the forward pass of LLMs: co-occurrence behavior and semantic pivot behavior. We attribute LLMs' two distinct behaviors to the co-occurrence frequency of words and find the semantic pivot from the pre-training dataset. Finally, to apply our findings to improve the cross-lingual ability of LLMs, we reconstruct a semantic pivot-aware pre-training dataset using documents with a high proportion of semantic pivots. Our experiments validate the effectiveness of our approach in enhancing cross-lingual ability. Our research contributes insights into the interpretability of LLMs and offers a method for improving LLMs' cross-lingual ability.