EMODIS: A Benchmark for Context-Dependent Emoji Disambiguation in Large Language Models

TL;DR

EMODIS is a benchmark designed to evaluate large language models' ability to interpret ambiguous emojis within minimal, contrastive textual contexts, revealing significant gaps in contextual understanding.

Contribution

This work introduces EMODIS, the first benchmark specifically targeting context-dependent emoji disambiguation in LLMs, and provides comprehensive evaluation results highlighting current limitations.

Findings

LLMs often fail to distinguish subtle contextual differences in emoji meanings.

Models show biases toward dominant interpretations of emojis.

There is a notable gap between human and LLM emoji understanding.

Abstract

Large language models (LLMs) are increasingly deployed in real-world communication settings, yet their ability to resolve context-dependent ambiguity remains underexplored. In this work, we present EMODIS, a new benchmark for evaluating LLMs' capacity to interpret ambiguous emoji expressions under minimal but contrastive textual contexts. Each instance in EMODIS comprises an ambiguous sentence containing an emoji, two distinct disambiguating contexts that lead to divergent interpretations, and a specific question that requires contextual reasoning. We evaluate both open-source and API-based LLMs, and find that even the strongest models frequently fail to distinguish meanings when only subtle contextual cues are present. Further analysis reveals systematic biases toward dominant interpretations and limited sensitivity to pragmatic contrast. EMODIS provides a rigorous testbed for assessing contextual disambiguation, and highlights the gap in semantic reasoning between humans and LLMs.