ACE: Action Concept Enhancement of Video-Language Models in Procedural Videos

TL;DR

This paper introduces ACE, a fine-tuning method that enhances video-language models' understanding of procedural actions by incorporating augmented synonyms and negatives, leading to better zero-shot classification in cooking and assembly videos.

Contribution

The paper proposes a novel fine-tuning technique, ACE, that improves VLMs' robustness and understanding of procedural actions through label augmentation and stochastic replacement.

Findings

Significant improvements in zero-shot action classification accuracy.

Enhanced embedding alignment of unseen action synonyms.

Maintained competitive performance on seen actions.

Abstract

Vision-language models (VLMs) are capable of recognizing unseen actions. However, existing VLMs lack intrinsic understanding of procedural action concepts. Hence, they overfit to fixed labels and are not invariant to unseen action synonyms. To address this, we propose a simple fine-tuning technique, Action Concept Enhancement (ACE), to improve the robustness and concept understanding of VLMs in procedural action classification. ACE continually incorporates augmented action synonyms and negatives in an auxiliary classification loss by stochastically replacing fixed labels during training. This creates new combinations of action labels over the course of fine-tuning and prevents overfitting to fixed action representations. We show the enhanced concept understanding of our VLM, by visualizing the alignment of encoded embeddings of unseen action synonyms in the embedding space. Our experiments on the ATA, IKEA and GTEA datasets demonstrate the efficacy of ACE in domains of cooking and assembly leading to significant improvements in zero-shot action classification while maintaining competitive performance on seen actions.