Team PyKale (xy9) Submission to the EPIC-Kitchens 2021 Unsupervised Domain Adaptation Challenge for Action Recognition

TL;DR

This paper presents a multi-modal transformer-based approach with adversarial domain adaptation for unsupervised action recognition in challenging EPIC-Kitchens videos, achieving competitive results.

Contribution

It introduces a novel multi-modal transformer framework with temporal attention and adversarial adaptation for unsupervised domain adaptation in complex video datasets.

Findings

Achieved top-5 accuracy in all tasks

Outperformed baseline methods on verb and noun classification

Secured 5th place in the challenge

Abstract

This report describes the technical details of our submission to the EPIC-Kitchens 2021 Unsupervised Domain Adaptation Challenge for Action Recognition. The EPIC-Kitchens dataset is more difficult than other video domain adaptation datasets due to multi-tasks with more modalities. Firstly, to participate in the challenge, we employ a transformer to capture the spatial information from each modality. Secondly, we employ a temporal attention module to model temporal-wise inter-dependency. Thirdly, we employ the adversarial domain adaptation network to learn the general features between labeled source and unlabeled target domain. Finally, we incorporate multiple modalities to improve the performance by a three-stream network with late fusion. Our network achieves the comparable performance with the state-of-the-art baseline T$A^3$N and outperforms the baseline on top-1 accuracy for verb class and top-5 accuracies for all three tasks which are verb, noun and action. Under the team name xy9, our submission achieved 5th place in terms of top-1 accuracy for verb class and all top-5 accuracies.