VideoPerceiver: Enhancing Fine-Grained Temporal Perception in Video Multimodal Large Language Models

TL;DR

VideoPerceiver is a new multimodal video language model that improves fine-grained temporal perception, especially for brief actions and rare events, through a two-stage training process involving contrastive learning and reinforcement learning.

Contribution

It introduces a novel training framework with key-information-missing video construction and a relative reward mechanism to enhance temporal sensitivity in video understanding.

Findings

Outperforms state-of-the-art models on fine-grained action benchmarks.

Effectively captures transient events in long videos.

Maintains strong performance on standard video-language tasks.

Abstract

We propose VideoPerceiver, a novel video multimodal large language model (VMLLM) that enhances fine-grained perception in video understanding, addressing VMLLMs' limited ability to reason about brief actions in short clips or rare transient events in long videos. VideoPerceiver adopts a two-stage training framework. During supervised fine-tuning (SFT), we construct "key-information-missing" videos by extracting event-action keywords from captions, identifying corresponding key frames, and replacing them with adjacent frames. We jointly encode original and modified video tokens with text tokens, aligning intermediate visual representations with keywords via an auxiliary contrastive loss to enhance sensitivity to fine-grained motion cues. In reinforcement learning (RL), both video variants are fed into the model to generate descriptions, and a novel relative reward ensures responses from complete videos outperform those from degraded inputs, explicitly training the model to recover temporally precise action details. We also curate a dataset of 80,000 videos with fine-grained actions and transient events. Experiments show VideoPerceiver substantially outperforms state-of-the-art VMLLMs on fine-grained action understanding and rare event captioning benchmarks, while maintaining strong performance on standard tasks. By prioritizing task-relevant visual features, our work redefines video-language model training for fine-grained perception.