Stands to Reason: Investigating the Effect of Reasoning on Idiomaticity Detection

TL;DR

This study investigates how reasoning capabilities in large language models influence idiomaticity detection, revealing that larger models benefit modestly from reasoning, while smaller models can improve with explicit definitions in prompts.

Contribution

It provides an empirical analysis of reasoning's impact on idiomaticity detection across different model sizes and explores prompt-based improvements for smaller models.

Findings

Larger models show better understanding of idiomatic expressions.

Chain-of-thought reasoning yields modest performance gains in larger models.

Providing definitions in prompts can improve smaller models' performance.

Abstract

The recent trend towards utilisation of reasoning models has improved the performance of Large Language Models (LLMs) across many tasks which involve logical steps. One linguistic task that could benefit from this framing is idiomaticity detection, as a potentially idiomatic expression must first be understood before it can be disambiguated and serves as a basis for reasoning. In this paper, we explore how reasoning capabilities in LLMs affect idiomaticity detection performance and examine the effect of model size. We evaluate, as open source representative models, the suite of DeepSeek-R1 distillation models ranging from 1.5B to 70B parameters across four idiomaticity detection datasets. We find the effect of reasoning to be smaller and more varied than expected. For smaller models, producing chain-of-thought (CoT) reasoning increases performance from Math-tuned intermediate models, but not to the levels of the base models, whereas larger models (14B, 32B, and 70B) show modest improvements. Our in-depth analyses reveal that larger models demonstrate good understanding of idiomaticity, successfully producing accurate definitions of expressions, while smaller models often fail to output the actual meaning. For this reason, we also experiment with providing definitions in the prompts of smaller models, which we show can improve performance in some cases.