Temporal Object-Aware Vision Transformer for Few-Shot Video Object Detection

TL;DR

This paper introduces a novel object-aware temporal modeling method for few-shot video object detection that enhances temporal consistency and detection accuracy without relying on complex region proposals, achieving significant performance improvements.

Contribution

It proposes a filtering-based temporal modeling approach that propagates high-confidence object features, reducing noise and improving detection in few-shot video scenarios without explicit object tube proposals.

Findings

Achieves AP improvements of 3.7% to 5.3% across multiple datasets in 5-shot setting.

Enhances temporal consistency without complex object proposals.

Demonstrates robustness across 1-shot, 3-shot, and 10-shot configurations.

Abstract

Few-shot Video Object Detection (FSVOD) addresses the challenge of detecting novel objects in videos with limited labeled examples, overcoming the constraints of traditional detection methods that require extensive training data. This task presents key challenges, including maintaining temporal consistency across frames affected by occlusion and appearance variations, and achieving novel object generalization without relying on complex region proposals, which are often computationally expensive and require task-specific training. Our novel object-aware temporal modeling approach addresses these challenges by incorporating a filtering mechanism that selectively propagates high-confidence object features across frames. This enables efficient feature progression, reduces noise accumulation, and enhances detection accuracy in a few-shot setting. By utilizing few-shot trained detection and classification heads with focused feature propagation, we achieve robust temporal consistency without depending on explicit object tube proposals. Our approach achieves performance gains, with AP improvements of 3.7% (FSVOD-500), 5.3% (FSYTV-40), 4.3% (VidOR), and 4.5 (VidVRD) in the 5-shot setting. Further results demonstrate improvements in 1-shot, 3-shot, and 10-shot configurations. We make the code public at: https://github.com/yogesh-iitj/fs-video-vit