DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition

TL;DR

This paper addresses the vulnerability of open-vocabulary action recognition models to noisy class descriptions by proposing a novel denoising framework that enhances robustness through iterative generation and discrimination, improving real-world applicability.

Contribution

The paper introduces DENOISER, a new framework that denoises noisy class descriptions and improves OVAR robustness, filling a gap in handling real-world noisy labels.

Findings

DENOISER significantly improves robustness against noisy class descriptions.

The iterative generation and discrimination process enhances classification accuracy.

Extensive experiments validate the effectiveness of the proposed method across datasets.

Abstract

As one of the fundamental video tasks in computer vision, Open-Vocabulary Action Recognition (OVAR) recently gains increasing attention, with the development of vision-language pre-trainings. To enable generalization of arbitrary classes, existing methods treat class labels as text descriptions, then formulate OVAR as evaluating embedding similarity between visual samples and textual classes. However, one crucial issue is completely ignored: the class descriptions given by users may be noisy, e.g., misspellings and typos, limiting the real-world practicality of vanilla OVAR. To fill the research gap, this paper pioneers to evaluate existing methods by simulating multi-level noises of various types, and reveals their poor robustness. To tackle the noisy OVAR task, we further propose one novel DENOISER framework, covering two parts: generation and discrimination. Concretely, the generative part denoises noisy class-text names via one decoding process, i.e., propose text candidates, then utilize inter-modal and intra-modal information to vote for the best. At the discriminative part, we use vanilla OVAR models to assign visual samples to class-text names, thus obtaining more semantics. For optimization, we alternately iterate between generative and discriminative parts for progressive refinements. The denoised text classes help OVAR models classify visual samples more accurately; in return, classified visual samples help better denoising. On three datasets, we carry out extensive experiments to show our superior robustness, and thorough ablations to dissect the effectiveness of each component.