Breaking Language Barriers: Equitable Performance in Multilingual Language Models

TL;DR

This paper introduces a method to improve multilingual language models' fairness by fine-tuning them on synthetic code-switched data, significantly enhancing low-resource language performance without sacrificing high-resource language capabilities.

Contribution

It presents a novel fine-tuning approach using synthetic code-switched datasets to address performance disparities in multilingual LLMs, ensuring more equitable language understanding.

Findings

Significant performance gains in low-resource languages after fine-tuning.

Performance preservation or improvement in high-resource languages.

Introduction of a new synthetic code-switched dataset for CSR tasks.

Abstract

Cutting-edge LLMs have emerged as powerful tools for multilingual communication and understanding. However, LLMs perform worse in Common Sense Reasoning (CSR) tasks when prompted in low-resource languages (LRLs) like Hindi or Swahili compared to high-resource languages (HRLs) like English. Equalizing this inconsistent access to quality LLM outputs is crucial to ensure fairness for speakers of LRLs and across diverse linguistic communities. In this paper, we propose an approach to bridge this gap in LLM performance. Our approach involves fine-tuning an LLM on synthetic code-switched text generated using controlled language-mixing methods. We empirically demonstrate that fine-tuning LLMs on synthetic code-switched datasets leads to substantial improvements in LRL model performance while preserving or enhancing performance in HRLs. Additionally, we present a new dataset of synthetic code-switched text derived from the CommonSenseQA dataset, featuring three distinct language ratio configurations.