Transformed ROIs for Capturing Visual Transformations in Videos

TL;DR

This paper introduces TROI, a module that enhances CNNs by explicitly modeling long-range spatial and temporal relationships in videos, leading to improved action recognition performance.

Contribution

The paper proposes TROI, a novel plug-and-play module for CNNs that directly relates and transforms mid-level feature regions in space and time for better video understanding.

Findings

Achieves state-of-the-art results on Something-Something-V2.

Outperforms existing methods on EPIC-Kitchens-100.

Effectively models long-range visual transformations in videos.

Abstract

Modeling the visual changes that an action brings to a scene is critical for video understanding. Currently, CNNs process one local neighbourhood at a time, thus contextual relationships over longer ranges, while still learnable, are indirect. We present TROI, a plug-and-play module for CNNs to reason between mid-level feature representations that are otherwise separated in space and time. The module relates localized visual entities such as hands and interacting objects and transforms their corresponding regions of interest directly in the feature maps of convolutional layers. With TROI, we achieve state-of-the-art action recognition results on the large-scale datasets Something-Something-V2 and EPIC-Kitchens-100.