Knowledge-Guided Short-Context Action Anticipation in Human-Centric Videos

TL;DR

This paper introduces a transformer-based approach enhanced with a symbolic knowledge graph to improve long-term human action anticipation in short video segments, outperforming existing methods on benchmark datasets.

Contribution

It presents a novel integration of symbolic knowledge graphs with transformers for action anticipation, boosting performance on long-term predictions from short video clips.

Findings

Outperforms state-of-the-art methods by up to 9% on benchmark datasets

Effective use of symbolic knowledge graphs enhances transformer attention mechanisms

Improves long-term action anticipation accuracy in human-centric videos

Abstract

This work focuses on anticipating long-term human actions, particularly using short video segments, which can speed up editing workflows through improved suggestions while fostering creativity by suggesting narratives. To this end, we imbue a transformer network with a symbolic knowledge graph for action anticipation in video segments by boosting certain aspects of the transformer's attention mechanism at run-time. Demonstrated on two benchmark datasets, Breakfast and 50Salads, our approach outperforms current state-of-the-art methods for long-term action anticipation using short video context by up to 9%.