Asynchronous Interaction Aggregation for Action Detection

TL;DR

The paper introduces the Asynchronous Interaction Aggregation network (AIA) for video action detection, which models diverse interactions efficiently and achieves state-of-the-art results on multiple datasets.

Contribution

It proposes the novel AIA network with Interaction Aggregation and Asynchronous Memory Update for improved long-term interaction modeling in action detection.

Findings

Achieved 3.7 mAP improvement on AVA dataset

Demonstrated effectiveness on UCF101-24 and EPIC-Kitchens datasets

Easy to train end-to-end with notable accuracy gains

Abstract

Understanding interaction is an essential part of video action detection. We propose the Asynchronous Interaction Aggregation network (AIA) that leverages different interactions to boost action detection. There are two key designs in it: one is the Interaction Aggregation structure (IA) adopting a uniform paradigm to model and integrate multiple types of interaction; the other is the Asynchronous Memory Update algorithm (AMU) that enables us to achieve better performance by modeling very long-term interaction dynamically without huge computation cost. We provide empirical evidence to show that our network can gain notable accuracy from the integrative interactions and is easy to train end-to-end. Our method reports the new state-of-the-art performance on AVA dataset, with 3.7 mAP gain (12.6% relative improvement) on validation split comparing to our strong baseline. The results on dataset UCF101-24 and EPIC-Kitchens further illustrate the effectiveness of our approach. Source code will be made public at: https://github.com/MVIG-SJTU/AlphAction .