Beyond Single Frames: Can LMMs Comprehend Temporal and Contextual Narratives in Image Sequences?

TL;DR

This paper introduces StripCipher, a new benchmark for evaluating Large Multimodal Models' ability to understand and reason over image sequences, revealing significant performance gaps compared to humans especially in reordering tasks.

Contribution

The paper presents StripCipher, a novel benchmark with a dataset and tasks for assessing LMMs' sequential image understanding, highlighting current limitations and challenges.

Findings

GPT-4o achieves 23.93% accuracy in reordering images

Performance gap of over 50% between LMMs and humans

Input format significantly affects LMMs' sequential reasoning

Abstract

Large Multimodal Models (LMMs) have achieved remarkable success across various visual-language tasks. However, existing benchmarks predominantly focus on single-image understanding, leaving the analysis of image sequences largely unexplored. To address this limitation, we introduce StripCipher, a comprehensive benchmark designed to evaluate capabilities of LMMs to comprehend and reason over sequential images. StripCipher comprises a human-annotated dataset and three challenging subtasks: visual narrative comprehension, contextual frame prediction, and temporal narrative reordering. Our evaluation of 16 state-of-the-art LMMs, including GPT-4o and Qwen2.5VL, reveals a significant performance gap compared to human capabilities, particularly in tasks that require reordering shuffled sequential images. For instance, GPT-4o achieves only 23.93% accuracy in the reordering subtask, which is 56.07% lower than human performance. Further quantitative analysis discuss several factors, such as input format of images, affecting the performance of LLMs in sequential understanding, underscoring the fundamental challenges that remain in the development of LMMs.