CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages

TL;DR

CodeMixBench is a comprehensive benchmark and synthetic data generation method designed to evaluate and improve large language models' ability to handle code-mixing across multiple languages and tasks.

Contribution

The paper introduces a new benchmark covering diverse tasks and languages, along with a novel synthetic data generation method for assessing LLMs' code-mixing capabilities.

Findings

LLMs underperform on code-mixed datasets across language families.

Increasing training data size and model scale may enhance performance.

Synthetic data generation effectively evaluates LLMs' code-mixing abilities.

Abstract

Code-mixing, the practice of switching between languages within a conversation, poses unique challenges for traditional NLP. Existing benchmarks are limited by their narrow language pairs and tasks, failing to adequately assess large language models' (LLMs) code-mixing abilities. Despite the recognized importance of code-mixing for multilingual users, research on LLMs in this context remains sparse. Additionally, current techniques for synthesizing code-mixed data are underdeveloped to generate code-mixing. In response, we introduce CodeMixBench, a comprehensive benchmark covering eight tasks, including three specific to LLMs and five traditional NLP tasks, and 18 languages across seven language families. We also propose a new method for generating large-scale synthetic code-mixed texts by combining word substitution with GPT-4 prompting. Our evaluation reveals consistent underperformance of LLMs on code-mixed datasets involving different language families. Enhancements in training data size, model scale, and few-shot learning could improve their performance. The code and dataset are available at https://github.com/Jeromeyluck/CodeMixBench.