Leveraging Foundation Models for Multimodal Graph-Based Action Recognition

TL;DR

This paper presents a novel multimodal graph-based framework that combines foundation models like VideoMAE and BERT with dynamic graph reasoning to improve fine-grained action recognition in videos.

Contribution

It introduces an adaptive, dynamic graph architecture integrating multimodal representations and a task-specific attention mechanism for enhanced action understanding.

Findings

Outperforms state-of-the-art baselines on benchmark datasets

Effectively captures spatiotemporal and semantic relationships

Demonstrates robustness and generalization in action recognition

Abstract

Foundation models have ushered in a new era for multimodal video understanding by enabling the extraction of rich spatiotemporal and semantic representations. In this work, we introduce a novel graph-based framework that integrates a vision-language foundation, leveraging VideoMAE for dynamic visual encoding and BERT for contextual textual embedding, to address the challenge of recognizing fine-grained bimanual manipulation actions. Departing from conventional static graph architectures, our approach constructs an adaptive multimodal graph where nodes represent frames, objects, and textual annotations, and edges encode spatial, temporal, and semantic relationships. These graph structures evolve dynamically based on learned interactions, allowing for flexible and context-aware reasoning. A task-specific attention mechanism within a Graph Attention Network further enhances this reasoning by modulating edge importance based on action semantics. Through extensive evaluations on diverse benchmark datasets, we demonstrate that our method consistently outperforms state-of-the-art baselines, underscoring the strength of combining foundation models with dynamic graph-based reasoning for robust and generalizable action recognition.