IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressions

TL;DR

IdioLink is a new benchmark that tests whether language models can connect idiomatic expressions to their literal or paraphrased meanings, revealing current models' limitations in semantic abstraction.

Contribution

The paper introduces IdioLink, a large retrieval benchmark for idiomatic expressions, highlighting the challenges models face in understanding idioms beyond surface form.

Findings

Current models struggle to retrieve equivalent meanings across idiomatic and literal expressions.

Embedding baselines rely on shallow semantic cues rather than true understanding.

IdioLink exposes significant gaps in idiom-aware semantic retrieval.

Abstract

Idioms pose a fundamental challenge for language models, as their meaning cannot be inferred from surface form alone. Understanding such expressions, therefore, requires semantic abstraction beyond lexical overlap. We introduce IdioLink, a retrieval benchmark designed to test whether models can link idiomatic expressions to conceptually equivalent meanings expressed in literal or paraphrased forms. IdioLink comprises 10,700 documents and 2,140 queries, spanning 107 idioms with both literal and figurative uses. Each document and query is annotated with spans that convey the core meaning. Evaluating strong embedding baselines (e.g., BGE, E5, Contriever, and Qwen), we show that current models struggle to retrieve equivalent meanings across divergent surface realizations, relying instead on topical and shallow semantic cues. IdioLink exposes key gaps in idiom-aware semantic retrieval and provides a challenging testbed for future models.