AssembleNet++: Assembling Modality Representations via Attention Connections

TL;DR

AssembleNet++ introduces a novel attention-based video model that dynamically integrates semantic object information with appearance and motion features, achieving state-of-the-art activity recognition performance without pre-training.

Contribution

The paper presents AssembleNet++, a new model with peer-attention that learns feature importance across modalities, improving existing architectures for video understanding.

Findings

Outperforms previous models on activity recognition datasets

Peer-attention effectively learns feature importance dynamically

Applicable to various architectures, enhancing their performance

Abstract

We create a family of powerful video models which are able to: (i) learn interactions between semantic object information and raw appearance and motion features, and (ii) deploy attention in order to better learn the importance of features at each convolutional block of the network. A new network component named peer-attention is introduced, which dynamically learns the attention weights using another block or input modality. Even without pre-training, our models outperform the previous work on standard public activity recognition datasets with continuous videos, establishing new state-of-the-art. We also confirm that our findings of having neural connections from the object modality and the use of peer-attention is generally applicable for different existing architectures, improving their performances. We name our model explicitly as AssembleNet++. The code will be available at: https://sites.google.com/corp/view/assemblenet/