ImageChain: Advancing Sequential Image-to-Text Reasoning in Multimodal Large Language Models

TL;DR

ImageChain enhances multimodal large language models with sequential reasoning over image sequences by modeling visual data as multi-turn conversations, significantly improving next-scene description accuracy and zero-shot out-of-domain performance.

Contribution

The paper introduces ImageChain, a novel framework that enables multimodal models to perform temporal reasoning over image sequences through multi-turn dialogue modeling.

Findings

Improves next-scene description accuracy from 3.7% to 19% in SimRate.

Achieves robust zero-shot performance across diverse domains.

Validates instruction-tuning in multimodal multi-turn conversations as essential for temporal reasoning.

Abstract

Reasoning over sequences of images remains a challenge for multimodal large language models (MLLMs). While recent models incorporate multi-image data during pre-training, they still struggle to recognize sequential structures, often treating images independently. This work introduces ImageChain, a framework that enhances MLLMs with sequential reasoning capabilities over image data by modeling visual sequences as a multi-turn conversation. In ImageChain, images are interleaved with corresponding textual descriptions to form a controlled dialogue that explicitly captures temporal dependencies and narrative progression. Our method optimizes for the task of next-scene description, where the model generates a context-aware description of an upcoming scene based on preceding visual and textual cues. We demonstrate that our approach improves performance on the next-scene description task -- achieving an average improvement from 3.7% to 19% in SimRate, a metric that quantifies semantic similarity to human-annotated ground truths. Moreover, ImageChain achieves robust zero-shot out-of-domain performance in applications ranging from comics to robotics. Extensive experiments validate that instruction-tuning in a multimodal, multi-turn conversation design is key to bridging the gap between static image understanding and temporally-aware reasoning.