Comparative Study of Multilingual Idioms and Similes in Large Language Models

TL;DR

This paper evaluates multilingual large language models on figurative language tasks, revealing performance variations across languages and figurative types, and highlighting the impact of prompt strategies and resource limitations.

Contribution

It provides a comprehensive comparison of LLMs on multilingual idiom and simile interpretation, including new Persian datasets and analysis of prompt engineering effects.

Findings

Prompt engineering improves performance but varies by language and figurative type.

Open-source models struggle with low-resource languages in simile interpretation.

Idiom interpretation approaches saturation, indicating need for more challenging tasks.

Abstract

This study addresses the gap in the literature concerning the comparative performance of LLMs in interpreting different types of figurative language across multiple languages. By evaluating LLMs using two multilingual datasets on simile and idiom interpretation, we explore the effectiveness of various prompt engineering strategies, including chain-of-thought, few-shot, and English translation prompts. We extend the language of these datasets to Persian as well by building two new evaluation sets. Our comprehensive assessment involves both closed-source (GPT-3.5, GPT-4o mini, Gemini 1.5), and open-source models (Llama 3.1, Qwen2), highlighting significant differences in performance across languages and figurative types. Our findings reveal that while prompt engineering methods are generally effective, their success varies by figurative type, language, and model. We also observe that open-source models struggle particularly with low-resource languages in similes. Additionally, idiom interpretation is nearing saturation for many languages, necessitating more challenging evaluations.