Leveraging Transformers for Weakly Supervised Object Localization in Unconstrained Videos

TL;DR

This paper introduces TrCAM-V, a transformer-based approach for weakly-supervised object localization in videos, utilizing pseudo-labels from CLIP and CRF loss to improve accuracy without bounding box annotations.

Contribution

The novel TrCAM-V method combines transformer architecture with pseudo-labels and CRF loss for enhanced weakly-supervised video object localization.

Findings

Achieves state-of-the-art localization accuracy on YouTube-Objects dataset.

Effectively uses pseudo-labels from CLIP for training without bounding boxes.

Real-time frame processing demonstrated during inference.

Abstract

Weakly-Supervised Video Object Localization (WSVOL) involves localizing an object in videos using only video-level labels, also referred to as tags. State-of-the-art WSVOL methods like Temporal CAM (TCAM) rely on class activation mapping (CAM) and typically require a pre-trained CNN classifier. However, their localization accuracy is affected by their tendency to minimize the mutual information between different instances of a class and exploit temporal information during training for downstream tasks, e.g., detection and tracking. In the absence of bounding box annotation, it is challenging to exploit precise information about objects from temporal cues because the model struggles to locate objects over time. To address these issues, a novel method called transformer based CAM for videos (TrCAM-V), is proposed for WSVOL. It consists of a DeiT backbone with two heads for classification and localization. The classification head is trained using standard classification loss (CL), while the localization head is trained using pseudo-labels that are extracted using a pre-trained CLIP model. From these pseudo-labels, the high and low activation values are considered to be foreground and background regions, respectively. Our TrCAM-V method allows training a localization network by sampling pseudo-pixels on the fly from these regions. Additionally, a conditional random field (CRF) loss is employed to align the object boundaries with the foreground map. During inference, the model can process individual frames for real-time localization applications. Extensive experiments on challenging YouTube-Objects unconstrained video datasets show that our TrCAM-V method achieves new state-of-the-art performance in terms of classification and localization accuracy.