Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection

TL;DR

This paper introduces OpenMixer, a novel method leveraging large vision-language models for open-vocabulary action detection in videos, enabling recognition and localization of both seen and unseen actions.

Contribution

The paper proposes OpenMixer, combining spatial and temporal blocks with dynamic alignment, to achieve open-vocabulary action detection using pre-trained VLMs within a query-based transformer framework.

Findings

OpenMixer outperforms baselines on detecting seen and unseen actions.

Established new OVAD benchmarks under various settings.

Demonstrated strong generalization from pre-trained VLMs.

Abstract

Action detection aims to detect (recognize and localize) human actions spatially and temporally in videos. Existing approaches focus on the closed-set setting where an action detector is trained and tested on videos from a fixed set of action categories. However, this constrained setting is not viable in an open world where test videos inevitably come beyond the trained action categories. In this paper, we address the practical yet challenging Open-Vocabulary Action Detection (OVAD) problem. It aims to detect any action in test videos while training a model on a fixed set of action categories. To achieve such an open-vocabulary capability, we propose a novel method OpenMixer that exploits the inherent semantics and localizability of large vision-language models (VLM) within the family of query-based detection transformers (DETR). Specifically, the OpenMixer is developed by spatial and temporal OpenMixer blocks (S-OMB and T-OMB), and a dynamically fused alignment (DFA) module. The three components collectively enjoy the merits of strong generalization from pre-trained VLMs and end-to-end learning from DETR design. Moreover, we established OVAD benchmarks under various settings, and the experimental results show that the OpenMixer performs the best over baselines for detecting seen and unseen actions. We release the codes, models, and dataset splits at https://github.com/Cogito2012/OpenMixer.