Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMs

TL;DR

This paper investigates the multilingual abilities of Large Language Models, revealing their translation-like behavior and performance disparities across languages, using a novel back-translation prompting method to analyze how training data influences multilingual competence.

Contribution

It introduces a systematic approach to evaluate multilingual performance disparities and explores how insufficient training data affects LLMs' cross-language generalization.

Findings

GPT exhibits translation-like behavior in multilingual tasks

Performance varies significantly across different languages

Back-translation prompts reveal insights into LLMs' multilingual capabilities

Abstract

Large Language Models (LLMs) have demonstrated exceptional natural language understanding abilities and have excelled in a variety of natural language processing (NLP)tasks in recent years. Despite the fact that most LLMs are trained predominantly in English, multiple studies have demonstrated their comparative performance in many other languages. However, fundamental questions persist regarding how LLMs acquire their multi-lingual abilities and how performance varies across different languages. These inquiries are crucial for the study of LLMs since users and researchers often come from diverse language backgrounds, potentially influencing their utilization and interpretation of LLMs' results. In this work, we propose a systematic way of qualifying the performance disparities of LLMs under multilingual settings. We investigate the phenomenon of across-language generalizations in LLMs, wherein insufficient multi-lingual training data leads to advanced multi-lingual capabilities. To accomplish this, we employ a novel back-translation-based prompting method. The results show that GPT exhibits highly translating-like behaviour in multilingual settings.