Code-Mixer Ya Nahi: Novel Approaches to Measuring Multilingual LLMs' Code-Mixing Capabilities

TL;DR

This paper introduces Rule-Based Prompting to evaluate and generate code-mixed sentences in multilingual LLMs, comparing their translation abilities across multiple language pairs and creating a code-mixed chatbot application.

Contribution

It proposes a novel Rule-Based Prompting technique for measuring and generating code-mixed sentences in multilingual LLMs, expanding evaluation methods beyond traditional k-shot prompting.

Findings

k-shot prompting often yields better results

Rule-Based prompting generates diverse code-mixed sentences

Multilingual LLMs show varying translation abilities in code-mixed contexts

Abstract

Multilingual Large Language Models (LLMs) have demonstrated exceptional performance in Machine Translation (MT) tasks. However, their MT abilities in the context of code-switching (the practice of mixing two or more languages in an utterance) remain under-explored. In this paper, we introduce Rule-Based Prompting, a novel prompting technique to generate code-mixed sentences. We measure and compare the code-mixed MT abilities of 3 popular multilingual LLMs: GPT-3.5-turbo, GPT-4, and Gemini Pro across five language pairs: English-{Hindi, Bengali, Gujarati, French, Spanish} using $k$-shot prompting ($k\in\{0, 1, 10, 20\}$) and Rule-Based Prompting. Our findings suggest that though $k$-shot prompting often leads to the best results, Rule-Based prompting shows promise in generating unique code-mixed sentences that vary in their style of code-mixing. We also use $k$-shot prompting to gauge the code-mixed to English translation abilities of multilingual LLMs. For this purpose, we create a gold-standard code-mixed dataset spanning five language pairs: English-{Hindi, Bengali, Gujarati, French, Spanish}. As a real-world application of our work, we create a code-mixed chatbot.