Adaptive Prototype Model for Attribute-based Multi-label Few-shot Action Recognition

TL;DR

The paper introduces the Adaptive Attribute Prototype Model (AAPM), a novel approach for multi-label few-shot action recognition that leverages textual constraints and attribute assignment to improve accuracy and robustness, validated on a new dataset.

Contribution

The paper proposes AAPM, incorporating TCM and AAM to enhance multi-label few-shot action recognition, and introduces the Multi-Kinetics dataset for evaluation.

Findings

AAPM achieves state-of-the-art results in multi-label few-shot action recognition.

The Text-Constrain Module effectively incorporates textual label information.

The Attribute Assignment Method improves training robustness.

Abstract

In real-world action recognition systems, incorporating more attributes helps achieve a more comprehensive understanding of human behavior. However, using a single model to simultaneously recognize multiple attributes can lead to a decrease in accuracy. In this work, we propose a novel method i.e. Adaptive Attribute Prototype Model (AAPM) for human action recognition, which captures rich action-relevant attribute information and strikes a balance between accuracy and robustness. Firstly, we introduce the Text-Constrain Module (TCM) to incorporate textual information from potential labels, and constrain the construction of different attributes prototype representations. In addition, we explore the Attribute Assignment Method (AAM) to address the issue of training bias and increase robustness during the training process.Furthermore, we construct a new video dataset with attribute-based multi-label called Multi-Kinetics for evaluation, which contains various attribute labels (e.g. action, scene, object, etc.) related to human behavior. Extensive experiments demonstrate that our AAPM achieves the state-of-the-art performance in both attribute-based multi-label few-shot action recognition and single-label few-shot action recognition. The project and dataset are available at an anonymous account https://github.com/theAAPM/AAPM