Reasoning Beyond Literal: Cross-style Multimodal Reasoning for Figurative Language Understanding

TL;DR

This paper introduces a framework for multimodal reasoning models that interpret figurative language, provide transparent reasoning, and generalize across styles, improving understanding of sarcasm, humor, and metaphors in vision-language tasks.

Contribution

It presents a three-step framework enabling models to interpret, reason about, and generalize across multiple figurative styles in multimodal settings, with improved performance and transparency.

Findings

Reasoning traces significantly enhance figurative understanding.

Knowledge transfers effectively between related styles like sarcasm and humor.

Joint training across styles yields a generalized model outperforming larger counterparts.

Abstract

Vision-language models (VLMs) have demonstrated strong reasoning abilities in literal multimodal tasks such as visual mathematics and science question answering. However, figurative language, such as sarcasm, humor, and metaphor, remains a significant challenge, as it conveys intent and emotion through subtle incongruities between expressed and intended meanings. In multimodal settings, accompanying images can amplify or invert textual meaning, demanding models that reason across modalities and account for subjectivity. We propose a three-step framework for developing efficient multimodal reasoning models that can (i) interpret multimodal figurative language, (ii) provide transparent reasoning traces, and (iii) generalize across multiple figurative styles. Experiments across four styles show that (1) incorporating reasoning traces substantially improves multimodal figurative understanding, (2) reasoning learned in one style can transfer to others, especially between related styles like sarcasm and humor, and (3) training jointly across styles yields a generalized reasoning VLM that outperforms much larger open- and closed-source models. Our findings show that lightweight VLMs with verifiable reasoning achieve robust cross-style generalization while providing inspectable reasoning traces for multimodal tasks. The code and implementation are available at https://github.com/scheshmi/CrossStyle-MMR.