Disentangling Language and Culture for Evaluating Multilingual Large Language Models

TL;DR

This paper proposes a Dual Evaluation Framework to assess multilingual LLMs by analyzing their performance across linguistic and cultural dimensions, revealing a cultural-linguistic synergy and neuron activation patterns that inform model evaluation.

Contribution

It introduces a novel framework for culturally and linguistically nuanced evaluation of multilingual LLMs, highlighting the importance of cultural context in model performance assessment.

Findings

Models perform better with culturally aligned questions.

Neuron activation correlates with cultural context in languages.

Cultural-linguistic synergy affects multilingual model performance.

Abstract

This paper introduces a Dual Evaluation Framework to comprehensively assess the multilingual capabilities of LLMs. By decomposing the evaluation along the dimensions of linguistic medium and cultural context, this framework enables a nuanced analysis of LLMs' ability to process questions within both native and cross-cultural contexts cross-lingually. Extensive evaluations are conducted on a wide range of models, revealing a notable "CulturalLinguistic Synergy" phenomenon, where models exhibit better performance when questions are culturally aligned with the language. This phenomenon is further explored through interpretability probing, which shows that a higher proportion of specific neurons are activated in a language's cultural context. This activation proportion could serve as a potential indicator for evaluating multilingual performance during model training. Our findings challenge the prevailing notion that LLMs, primarily trained on English data, perform uniformly across languages and highlight the necessity of culturally and linguistically model evaluations. Our code can be found at https://yingjiahao14. github.io/Dual-Evaluation/.