MangaUB: A Manga Understanding Benchmark for Large Multimodal Models

TL;DR

MangaUB is a new benchmark designed to evaluate large multimodal models' ability to understand manga, highlighting strengths in image recognition and challenges in cross-panel comprehension.

Contribution

The paper introduces MangaUB, a comprehensive benchmark for assessing LMMs' manga understanding capabilities across single and multiple panels.

Findings

Strong performance in image content recognition

Challenges in understanding emotions across panels

Identifies areas for future model improvements

Abstract

Manga is a popular medium that combines stylized drawings and text to convey stories. As manga panels differ from natural images, computational systems traditionally had to be designed specifically for manga. Recently, the adaptive nature of modern large multimodal models (LMMs) shows possibilities for more general approaches. To provide an analysis of the current capability of LMMs for manga understanding tasks and identifying areas for their improvement, we design and evaluate MangaUB, a novel manga understanding benchmark for LMMs. MangaUB is designed to assess the recognition and understanding of content shown in a single panel as well as conveyed across multiple panels, allowing for a fine-grained analysis of a model's various capabilities required for manga understanding. Our results show strong performance on the recognition of image content, while understanding the emotion and information conveyed across multiple panels is still challenging, highlighting future work towards LMMs for manga understanding.