Graph-Based Multimodal and Multi-view Alignment for Keystep Recognition

TL;DR

This paper introduces a graph-based framework for keystep recognition in egocentric videos, leveraging long-term dependencies and cross-view alignment to improve accuracy and efficiency.

Contribution

It presents a novel flexible graph-learning approach that models video clips as nodes, incorporating multimodal features and cross-view alignment for enhanced keystep recognition.

Findings

Outperforms existing methods by over 12% in accuracy

Constructed graphs are sparse and computationally efficient

Effectively leverages multimodal features like narrations, depth, and object labels

Abstract

Egocentric videos capture scenes from a wearer's viewpoint, resulting in dynamic backgrounds, frequent motion, and occlusions, posing challenges to accurate keystep recognition. We propose a flexible graph-learning framework for fine-grained keystep recognition that is able to effectively leverage long-term dependencies in egocentric videos, and leverage alignment between egocentric and exocentric videos during training for improved inference on egocentric videos. Our approach consists of constructing a graph where each video clip of the egocentric video corresponds to a node. During training, we consider each clip of each exocentric video (if available) as additional nodes. We examine several strategies to define connections across these nodes and pose keystep recognition as a node classification task on the constructed graphs. We perform extensive experiments on the Ego-Exo4D dataset and show that our proposed flexible graph-based framework notably outperforms existing methods by more than 12 points in accuracy. Furthermore, the constructed graphs are sparse and compute efficient. We also present a study examining on harnessing several multimodal features, including narrations, depth, and object class labels, on a heterogeneous graph and discuss their corresponding contribution to the keystep recognition performance.