Two-stream joint matching method based on contrastive learning for few-shot action recognition

TL;DR

This paper introduces a two-stream joint matching method using contrastive learning to improve few-shot action recognition by better modeling action relations and handling video matching challenges.

Contribution

It proposes a novel two-module approach combining multi-modal contrastive learning and joint matching to address existing limitations in few-shot action recognition.

Findings

Enhanced action relation modeling through multi-modal contrastive learning.

Improved handling of variable-length and misaligned videos.

Validated effectiveness on SSv2 and Kinetics datasets.

Abstract

Although few-shot action recognition based on metric learning paradigm has achieved significant success, it fails to address the following issues: (1) inadequate action relation modeling and underutilization of multi-modal information; (2) challenges in handling video matching problems with different lengths and speeds, and video matching problems with misalignment of video sub-actions. To address these issues, we propose a Two-Stream Joint Matching method based on contrastive learning (TSJM), which consists of two modules: Multi-modal Contrastive Learning Module (MCL) and Joint Matching Module (JMM). The objective of the MCL is to extensively investigate the inter-modal mutual information relationships, thereby thoroughly extracting modal information to enhance the modeling of action relationships. The JMM aims to simultaneously address the aforementioned video matching problems. The effectiveness of the proposed method is evaluated on two widely used few shot action recognition datasets, namely, SSv2 and Kinetics. Comprehensive ablation experiments are also conducted to substantiate the efficacy of our proposed approach.