IPFormer-VideoLLM: Enhancing Multi-modal Video Understanding for Multi-shot Scenes

TL;DR

This paper introduces IPFormer-VideoLLM, a new model with instance-level prompts and a dataset, MultiClip-Bench, to improve multi-shot video understanding, addressing challenges like scene changes and camera angles.

Contribution

The work presents a novel dataset for multi-shot scenarios and a new model that effectively encodes instance-specific information across scenes.

Findings

Enhanced multi-shot performance on new dataset

Improved accuracy across various video benchmarks

Effective encoding of instance-level features

Abstract

Video Large Language Models (VideoLLMs) have demonstrated remarkable understanding capabilities, but are found struggling to tackle multi-shot scenarios,e.g., video clips with varying camera angles or scene changes. This challenge can render failures such as instance identity forgetting and key frame negligence. In this work, we first attribute the challenge to the lack of multi-shot annotations among existing datasets and therefore we introduce a new dataset termed MultiClip-Bench, featuring dense descriptions and instruction-based question-answering pairs tailored for multi-shot scenarios. We empirically find that the training set significantly boosts the multi-shot performance, while the testing benchmark provides a reliable measure of the model capability in multi-shot scenarios. By further analyzing and discovering that current models only encode instance features in a discrete or lossy manner, at the risk of missing identity information, we then contribute a new model IPFormer-VideoLLM. Its key idea is the injection of instance-level features as instance prompts through an efficient attention-based connector. This allows for the aggregation of instance-specific information across scenes. Experiments demonstrate that our proposed dataset and model not only enhance the multi-scene video understanding significantly, but also offer distinct advantages across various video benchmarks.