Multimodal UNcommonsense: From Odd to Ordinary and Ordinary to Odd

TL;DR

This paper introduces MUN, a benchmark for evaluating multimodal commonsense reasoning in atypical scenarios, and proposes R-ICL with MER to improve model performance without extra training.

Contribution

The paper presents a new benchmark, MUN, and a retrieval-based in-context learning framework, R-ICL, to enhance reasoning in unusual multimodal contexts without additional training.

Findings

R-ICL improves performance by 8.3% over baseline methods.

MER effectively retrieves relevant exemplars in discordant multimodal pairs.

MUN enables evaluation of model robustness in diverse, non-typical scenarios.

Abstract

Commonsense reasoning in multimodal contexts remains a foundational challenge in artificial intelligence. We introduce Multimodal UNcommonsense(MUN), a benchmark designed to evaluate models' ability to handle scenarios that deviate from typical visual or contextual expectations. MUN pairs visual scenes with surprising or unlikely outcomes described in natural language, prompting models to either rationalize seemingly odd images using everyday logic or uncover unexpected interpretations in ordinary scenes. To support this task, we propose a retrieval-based in-context learning (R-ICL) framework that transfers reasoning capabilities from larger models to smaller ones without additional training. Leveraging a novel Multimodal Ensemble Retriever (MER), our method identifies semantically relevant exemplars even when image and text pairs are deliberately discordant. Experiments show an average improvement of 8.3% over baseline ICL methods, highlighting the effectiveness of R-ICL in low-frequency, atypical settings. MUN opens new directions for evaluating and improving visual-language models' robustness and adaptability in real-world, culturally diverse, and non-prototypical scenarios.