Structured Video Tokens @ Ego4D PNR Temporal Localization Challenge 2022

TL;DR

This paper introduces StructureViT, a transformer-based framework that leverages structured image information and cross-modal consistency to improve temporal localization in egocentric videos, achieving strong challenge results.

Contribution

The novel SViT framework utilizes object tokens and scene alignment to enhance video understanding using limited training images.

Findings

Achieved 0.656 absolute temporal localization error on the challenge test set.

Demonstrated effective use of image structure to improve video temporal localization.

Proposed a Frame-Clip Consistency loss for cross-modal alignment.

Abstract

This technical report describes the SViT approach for the Ego4D Point of No Return (PNR) Temporal Localization Challenge. We propose a learning framework StructureViT (SViT for short), which demonstrates how utilizing the structure of a small number of images only available during training can improve a video model. SViT relies on two key insights. First, as both images and videos contain structured information, we enrich a transformer model with a set of \emph{object tokens} that can be used across images and videos. Second, the scene representations of individual frames in video should "align" with those of still images. This is achieved via a "Frame-Clip Consistency" loss, which ensures the flow of structured information between images and videos. SViT obtains strong performance on the challenge test set with 0.656 absolute temporal localization error.