Re:Verse -- Can Your VLM Read a Manga?

TL;DR

This paper investigates the limitations of current Vision Language Models in understanding manga narratives, highlighting their struggles with temporal reasoning, character consistency, and story coherence across extended sequences.

Contribution

It introduces a novel evaluation framework combining multimodal annotation, cross-modal analysis, and retrieval methods to systematically assess narrative understanding in VLMs.

Findings

Current models excel at panel interpretation but fail at causal and temporal reasoning.

The framework applied to Re:Zero manga reveals significant gaps in story-level comprehension.

Provides actionable insights and a foundation for future narrative intelligence evaluation.

Abstract

Current Vision Language Models (VLMs) demonstrate a critical gap between surface-level recognition and deep narrative reasoning when processing sequential visual storytelling. Through a comprehensive investigation of manga narrative understanding, we reveal that while recent large multimodal models excel at individual panel interpretation, they systematically fail at temporal causality and cross-panel cohesion, core requirements for coherent story comprehension. We introduce a novel evaluation framework that combines fine-grained multimodal annotation, cross-modal embedding analysis, and retrieval-augmented assessment to systematically characterize these limitations. Our methodology includes (i) a rigorous annotation protocol linking visual elements to narrative structure through aligned light novel text, (ii) comprehensive evaluation across multiple reasoning paradigms, including direct inference and retrieval-augmented generation, and (iii) cross-modal similarity analysis revealing fundamental misalignments in current VLMs' joint representations. Applying this framework to Re:Zero manga across 11 chapters with 308 annotated panels, we conduct the first systematic study of long-form narrative understanding in VLMs through three core evaluation axes: generative storytelling, contextual dialogue grounding, and temporal reasoning. Our findings demonstrate that current models lack genuine story-level intelligence, struggling particularly with non-linear narratives, character consistency, and causal inference across extended sequences. This work establishes both the foundation and practical methodology for evaluating narrative intelligence, while providing actionable insights into the capability of deep sequential understanding of Discrete Visual Narratives beyond basic recognition in Multimodal Models. Project Page: https://re-verse.vercel.app