IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Model

TL;DR

This paper introduces IDA-VLM, a large vision-language model designed to improve understanding of complex visual narratives like movies by recognizing and associating character identities across multiple scenes, and presents a new benchmark MM-ID.

Contribution

The paper proposes a novel ID-aware vision-language model and a benchmark for instance ID recognition, addressing a key limitation in current models for movie understanding.

Findings

Existing LVLMs struggle with instance ID recognition across scenes.

IDA-VLM improves character identity association in visual narratives.

The MM-ID benchmark evaluates multi-identity recognition capabilities.

Abstract

The rapid advancement of Large Vision-Language models (LVLMs) has demonstrated a spectrum of emergent capabilities. Nevertheless, current models only focus on the visual content of a single scenario, while their ability to associate instances across different scenes has not yet been explored, which is essential for understanding complex visual content, such as movies with multiple characters and intricate plots. Towards movie understanding, a critical initial step for LVLMs is to unleash the potential of character identities memory and recognition across multiple visual scenarios. To achieve the goal, we propose visual instruction tuning with ID reference and develop an ID-Aware Large Vision-Language Model, IDA-VLM. Furthermore, our research introduces a novel benchmark MM-ID, to examine LVLMs on instance IDs memory and recognition across four dimensions: matching, location, question-answering, and captioning. Our findings highlight the limitations of existing LVLMs in recognizing and associating instance identities with ID reference. This paper paves the way for future artificial intelligence systems to possess multi-identity visual inputs, thereby facilitating the comprehension of complex visual narratives like movies.