M&M Mix: A Multimodal Multiview Transformer Ensemble

TL;DR

This paper presents M&M Mix, an ensemble of multimodal multiview transformers for action recognition, achieving state-of-the-art results by adapting and combining multiple MTV models with different modalities and backbone sizes.

Contribution

It introduces a multimodal multiview transformer ensemble that significantly improves action recognition accuracy over previous methods.

Findings

Achieved 52.8% Top-1 accuracy on Epic-Kitchens test set.

Outperformed last year's winning entry by 4.1%.

Demonstrated effectiveness of multimodal ensemble approach.

Abstract

This report describes the approach behind our winning solution to the 2022 Epic-Kitchens Action Recognition Challenge. Our approach builds upon our recent work, Multiview Transformer for Video Recognition (MTV), and adapts it to multimodal inputs. Our final submission consists of an ensemble of Multimodal MTV (M&M) models varying backbone sizes and input modalities. Our approach achieved 52.8% Top-1 accuracy on the test set in action classes, which is 4.1% higher than last year's winning entry.