AbductiveMLLM: Boosting Visual Abductive Reasoning Within MLLMs

TL;DR

AbductiveMLLM enhances visual abductive reasoning in multimodal large language models by integrating verbal hypothesis pruning and pictorial scene imagining, leading to state-of-the-art results on VAR benchmarks.

Contribution

This paper introduces a novel dual-mode framework with REASONER and IMAGINER components to improve abductive inference in MLLMs, inspired by human cognition.

Findings

Achieves state-of-the-art performance on VAR benchmarks.

Outperforms traditional solutions and advanced MLLMs.

Demonstrates effective integration of verbal and pictorial reasoning.

Abstract

Visual abductive reasoning (VAR) is a challenging task that requires AI systems to infer the most likely explanation for incomplete visual observations. While recent MLLMs develop strong general-purpose multimodal reasoning capabilities, they fall short in abductive inference, as compared to human beings. To bridge this gap, we draw inspiration from the interplay between verbal and pictorial abduction in human cognition, and propose to strengthen abduction of MLLMs by mimicking such dual-mode behavior. Concretely, we introduce AbductiveMLLM comprising of two synergistic components: REASONER and IMAGINER. The REASONER operates in the verbal domain. It first explores a broad space of possible explanations using a blind LLM and then prunes visually incongruent hypotheses based on cross-modal causal alignment. The remaining hypotheses are introduced into the MLLM as targeted priors, steering its reasoning toward causally coherent explanations. The IMAGINER, on the other hand, further guides MLLMs by emulating human-like pictorial thinking. It conditions a text-to-image diffusion model on both the input video and the REASONER's output embeddings to "imagine" plausible visual scenes that correspond to verbal explanation, thereby enriching MLLMs' contextual grounding. The two components are trained jointly in an end-to-end manner. Experiments on standard VAR benchmarks show that AbductiveMLLM achieves state-of-the-art performance, consistently outperforming traditional solutions and advanced MLLMs.