Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer

TL;DR

This paper introduces SALT, a novel method for cross-lingual transfer that recycles embeddings from target language PLMs to improve multilingual LLMs, outperforming existing approaches in understanding tasks.

Contribution

SALT is a new transfer technique that uses regression lines based on vocabulary overlap to better utilize target language embeddings in LLMs.

Findings

SALT outperforms other transfer methods in cross-lingual tasks.

SALT achieves faster convergence during language adaptation.

SALT enhances LLM performance across various architectures.

Abstract

Large Language Models (LLMs) increasingly incorporate multilingual capabilities, fueling the demand to transfer them into target language-specific models. However, most approaches, which blend the source model's embedding by replacing the source vocabulary with the target language-specific vocabulary, may constrain expressive capacity in the target language since the source model is predominantly trained on English data. In this paper, we propose Semantic Aware Linear Transfer (SALT), a novel cross-lingual transfer technique that recycles embeddings from target language Pre-trained Language Models (PLMs) to transmit the deep representational strengths of PLM-derived embedding to LLMs. SALT derives unique regression lines based on the similarity in the overlap of the source and target vocabularies, to handle each non-overlapping token's embedding space. Our extensive experiments show that SALT significantly outperforms other transfer methods and achieves lower loss with accelerating faster convergence during language adaptation. Notably, SALT obtains remarkable performance in cross-lingual understanding setups compared to other methods. Furthermore, we highlight the scalable use of PLMs to enhance the functionality of contemporary LLMs by conducting experiments with varying architectures.