# Representation Learning on Visual-Symbolic Graphs for Video Understanding

## TL;DR

This paper introduces a hybrid graph neural network that combines visual and symbolic graphs to improve video understanding, capturing complex interactions and semantic relationships for better action localization.

## Contribution

It presents a novel hybrid graph neural network framework that integrates visual and symbolic graphs with specialized attention mechanisms for enhanced video analysis.

## Key findings

- Achieved state-of-the-art results on Charades dataset.
- Effectively models complex actor-object interactions.
- Improves temporal action localization accuracy.

## Abstract

Events in natural videos typically arise from spatio-temporal interactions between actors and objects and involve multiple co-occurring activities and object classes. To capture this rich visual and semantic context, we propose using two graphs: (1) an attributed spatio-temporal visual graph whose nodes correspond to actors and objects and whose edges encode different types of interactions, and (2) a symbolic graph that models semantic relationships. We further propose a graph neural network for refining the representations of actors, objects and their interactions on the resulting hybrid graph. Our model goes beyond current approaches that assume nodes and edges are of the same type, operate on graphs with fixed edge weights and do not use a symbolic graph. In particular, our framework: a) has specialized attention-based message functions for different node and edge types; b) uses visual edge features; c) integrates visual evidence with label relationships; and d) performs global reasoning in the semantic space. Experiments on challenging video understanding tasks, such as temporal action localization on the Charades dataset, show that the proposed method leads to state-of-the-art performance.