Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings

TL;DR

This paper investigates whether multilingual large language models can culturally-diverse reasoners by evaluating their understanding of proverbs and sayings across different languages and cultures, revealing limitations and a cultural gap.

Contribution

The study introduces MAPS, a new dataset for evaluating multilingual LLMs on proverb understanding in conversational contexts, highlighting their limited reasoning about figurative language and cultural differences.

Findings

LLMs have limited knowledge of proverbs and do not truly understand them.

Struggle of LLMs to reason with figurative proverbs and sayings.

Existence of a 'culture gap' in LLM reasoning across languages.

Abstract

Large language models (LLMs) are highly adept at question answering and reasoning tasks, but when reasoning in a situational context, human expectations vary depending on the relevant cultural common ground. As languages are associated with diverse cultures, LLMs should also be culturally-diverse reasoners. In this paper, we study the ability of a wide range of state-of-the-art multilingual LLMs (mLLMs) to reason with proverbs and sayings in a conversational context. Our experiments reveal that: (1) mLLMs "know" limited proverbs and memorizing proverbs does not mean understanding them within a conversational context; (2) mLLMs struggle to reason with figurative proverbs and sayings, and when asked to select the wrong answer (instead of asking it to select the correct answer); and (3) there is a "culture gap" in mLLMs when reasoning about proverbs and sayings translated from other languages. We construct and release our evaluation dataset MAPS (MulticultrAl Proverbs and Sayings) for proverb understanding with conversational context for six different languages.