ExpressivityBench: Can LLMs Communicate Implicitly?

TL;DR

ExpressivityBench evaluates how well large language models can communicate implicit information like emotion and identity, revealing strengths in affective expression but weaknesses in sociolinguistic signals, with implications for socially-aware AI applications.

Contribution

Introduces a novel benchmark for assessing LLMs' implicit communication abilities using information-theoretic models and scalable LLM-based human validation.

Findings

Models communicate affective content effectively.

Models struggle with sociolinguistic signals.

Human baselines outperform models in implicit communication.

Abstract

Human communication is often implicit, conveying tone, identity, and intent beyond literal meanings. While large language models have achieved strong performance on explicit tasks such as summarization and reasoning, their capacity for expressivity, or implicit communication, remains underexplored. We introduce \textbf{ExpressivityBench}, a framework for evaluating the expressivity of LLMs using information-theoretic communication models. Our approach quantifies how well LLM-generated text communicates target properties without explicit mention, across nine tasks spanning emotion, identity, and tone. To enable scalable and reproducible evaluation, we employ LLM-based graders validated against human judgments. Our results reveal that while models are adept at expressing affective content, they struggle with sociolinguistic signals, lagging behind human baselines. This study provides a necessary step to evaluate human-like implicit communication, with implications for applications such as education, mental health support, and socially-aware dialogue systems. We provide code and data for our benchmark alongside our paper.