MERaLiON-TextLLM: Cross-Lingual Understanding of Large Language Models in Chinese, Indonesian, Malay, and Singlish

TL;DR

MERaLiON-TextLLM introduces open-source multilingual models tailored for Chinese, Indonesian, Malay, and Singlish, enhancing cross-lingual understanding and outperforming baseline Llama-3 models through continued pre-training and weight merging.

Contribution

The paper presents a new series of open-source multilingual language models specifically optimized for underrepresented languages, with improved performance over existing models.

Findings

Performance improvements across multiple benchmarks in target languages.

Model checkpoints provided as resources for further research.

Enhanced understanding and generation capabilities in Chinese, Indonesian, Malay, and Singlish.

Abstract

Multilingual large language models (MLLMs) have shown impressive capabilities across a variety of languages. However, efficacy can differ greatly between different language families, especially for those with limited linguistic resources. This report presents MERaLiON-TextLLM, a series of open-source language models specifically tailored to improve understanding and generation in Chinese, Indonesian, Malay, and Singlish. The initial released model is built on Llama-3-8B-Base and refined through a meticulously crafted process of continued pre-training and weight merging. Our approach achieves performance improvements across benchmarks in these languages, exceeding the capabilities of the official Llama-3 models. We provide the model checkpoints as a resource to support further research and development in cross-lingual language understanding.