Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognition

TL;DR

This paper introduces AMeFu-Net, a novel depth-guided meta-fusion network that enhances few-shot video recognition by integrating depth information, feature-level augmentation, and adaptive fusion within a meta-learning framework.

Contribution

The paper proposes a depth-guided adaptive fusion approach with a new DGAdaIN module and meta-learning training for improved few-shot video recognition.

Findings

Effective in few-shot video recognition benchmarks

Depth information improves recognition accuracy

Meta-learning enhances model adaptability

Abstract

Humans can easily recognize actions with only a few examples given, while the existing video recognition models still heavily rely on the large-scale labeled data inputs. This observation has motivated an increasing interest in few-shot video action recognition, which aims at learning new actions with only very few labeled samples. In this paper, we propose a depth guided Adaptive Meta-Fusion Network for few-shot video recognition which is termed as AMeFu-Net. Concretely, we tackle the few-shot recognition problem from three aspects: firstly, we alleviate this extremely data-scarce problem by introducing depth information as a carrier of the scene, which will bring extra visual information to our model; secondly, we fuse the representation of original RGB clips with multiple non-strictly corresponding depth clips sampled by our temporal asynchronization augmentation mechanism, which synthesizes new instances at feature-level; thirdly, a novel Depth Guided Adaptive Instance Normalization (DGAdaIN) fusion module is proposed to fuse the two-stream modalities efficiently. Additionally, to better mimic the few-shot recognition process, our model is trained in the meta-learning way. Extensive experiments on several action recognition benchmarks demonstrate the effectiveness of our model.