Task-Specific Alignment and Multiple Level Transformer for Few-Shot Action Recognition

TL;DR

This paper introduces TSA-MLT, an end-to-end framework for few-shot action recognition that filters irrelevant frames and leverages multi-level features with a fusion loss, achieving state-of-the-art results on multiple datasets.

Contribution

The paper proposes TSA-MLT, a novel method combining task-specific frame filtering and multi-level feature fusion with a fusion loss for improved few-shot video action recognition.

Findings

Achieves state-of-the-art results on HMDB51 and UCF101 datasets.

Demonstrates competitive performance on Kinetics and Something-Something V2 datasets.

Effectively filters irrelevant frames and exploits multi-level features for better recognition.

Abstract

In the research field of few-shot learning, the main difference between image-based and video-based is the additional temporal dimension. In recent years, some works have used the Transformer to deal with frames, then get the attention feature and the enhanced prototype, and the results are competitive. However, some video frames may relate little to the action, and only using single frame-level or segment-level features may not mine enough information. We address these problems sequentially through an end-to-end method named "Task-Specific Alignment and Multiple-level Transformer Network (TSA-MLT)". The first module (TSA) aims at filtering the action-irrelevant frames for action duration alignment. Affine Transformation for frame sequence in the time dimension is used for linear sampling. The second module (MLT) focuses on the Multiple-level feature of the support prototype and query sample to mine more information for the alignment, which operates on different level features. We adopt a fusion loss according to a fusion distance that fuses the L2 sequence distance, which focuses on temporal order alignment, and the Optimal Transport distance, which focuses on measuring the gap between the appearance and semantics of the videos. Extensive experiments show our method achieves state-of-the-art results on the HMDB51 and UCF101 datasets and a competitive result on the benchmark of Kinetics and something 2-something V2 datasets. Our code is available at the URL: https://github.com/cofly2014/tsa-mlt.git