Cut-Based Graph Learning Networks to Discover Compositional Structure of Sequential Video Data

TL;DR

This paper introduces Cut-Based Graph Learning Networks (CB-GLNs) that model complex, variable-length semantic dependencies in videos by representing them as graphs, outperforming traditional sequential models.

Contribution

The paper proposes a novel graph-based framework for video data that captures complex dependencies through multilevel graph structures and a parameterized kernel with graph-cut and message passing.

Findings

CB-GLNs outperform baseline methods in video classification and Q&A tasks.

The model effectively captures semantic compositional structures in videos.

Experimental results demonstrate improved accuracy on Youtube-8M and TVQA datasets.

Abstract

Conventional sequential learning methods such as Recurrent Neural Networks (RNNs) focus on interactions between consecutive inputs, i.e. first-order Markovian dependency. However, most of sequential data, as seen with videos, have complex dependency structures that imply variable-length semantic flows and their compositions, and those are hard to be captured by conventional methods. Here, we propose Cut-Based Graph Learning Networks (CB-GLNs) for learning video data by discovering these complex structures of the video. The CB-GLNs represent video data as a graph, with nodes and edges corresponding to frames of the video and their dependencies respectively. The CB-GLNs find compositional dependencies of the data in multilevel graph forms via a parameterized kernel with graph-cut and a message passing framework. We evaluate the proposed method on the two different tasks for video understanding: Video theme classification (Youtube-8M dataset) and Video Question and Answering (TVQA dataset). The experimental results show that our model efficiently learns the semantic compositional structure of video data. Furthermore, our model achieves the highest performance in comparison to other baseline methods.