Training-Free Zero-Shot Temporal Action Detection with Vision-Language Models

TL;DR

This paper introduces a training-free, zero-shot temporal action detection method using vision-language models, achieving competitive results without additional training or fine-tuning.

Contribution

The paper proposes a novel training-free approach for zero-shot action detection that leverages vision-language models and introduces new scoring and adaptation strategies.

Findings

Outperforms state-of-the-art unsupervised methods on THUMOS14 and ActivityNet-1.3.

Requires only 1/13 of the runtime of comparable methods.

Test-time adaptation improves detection performance significantly.

Abstract

Existing zero-shot temporal action detection (ZSTAD) methods predominantly use fully supervised or unsupervised strategies to recognize unseen activities. However, these training-based methods are prone to domain shifts and require high computational costs, which hinder their practical applicability in real-world scenarios. In this paper, unlike previous works, we propose a training-Free Zero-shot temporal Action Detection (FreeZAD) method, leveraging existing vision-language (ViL) models to directly classify and localize unseen activities within untrimmed videos without any additional fine-tuning or adaptation. We mitigate the need for explicit temporal modeling and reliance on pseudo-label quality by designing the LOGarithmic decay weighted Outer-Inner-Contrastive Score (LogOIC) and frequency-based Actionness Calibration. Furthermore, we introduce a test-time adaptation (TTA) strategy using Prototype-Centric Sampling (PCS) to expand FreeZAD, enabling ViL models to adapt more effectively for ZSTAD. Extensive experiments on the THUMOS14 and ActivityNet-1.3 datasets demonstrate that our training-free method outperforms state-of-the-art unsupervised methods while requiring only 1/13 of the runtime. When equipped with TTA, the enhanced method further narrows the gap with fully supervised methods.