Text-driven Online Action Detection

TL;DR

This paper introduces TOAD, a novel text-driven online action detection model utilizing CLIP embeddings, achieving high accuracy and supporting zero-shot and few-shot learning with reduced computational costs.

Contribution

The paper presents TOAD, the first architecture to leverage vision-language models for online action detection, enabling efficient zero-shot and few-shot learning.

Findings

Achieves 82.46% mAP on THUMOS14 dataset.

Outperforms existing methods in zero-shot and few-shot settings.

Sets new benchmarks for online action detection performance.

Abstract

Detecting actions as they occur is essential for applications like video surveillance, autonomous driving, and human-robot interaction. Known as online action detection, this task requires classifying actions in streaming videos, handling background noise, and coping with incomplete actions. Transformer architectures are the current state-of-the-art, yet the potential of recent advancements in computer vision, particularly vision-language models (VLMs), remains largely untapped for this problem, partly due to high computational costs. In this paper, we introduce TOAD: a Text-driven Online Action Detection architecture that supports zero-shot and few-shot learning. TOAD leverages CLIP (Contrastive Language-Image Pretraining) textual embeddings, enabling efficient use of VLMs without significant computational overhead. Our model achieves 82.46% mAP on the THUMOS14 dataset, outperforming existing methods, and sets new baselines for zero-shot and few-shot performance on the THUMOS14 and TVSeries datasets.