STAR-GNN: Spatial-Temporal Video Representation for Content-based Retrieval

TL;DR

STAR-GNN introduces a graph neural network-based framework for video feature representation that captures spatial and temporal dynamics, improving content-based video retrieval accuracy and robustness.

Contribution

It presents a novel multi-scale lattice feature graph model with a pluggable GNN component trained with triplet loss for enhanced video retrieval.

Findings

Achieves state-of-the-art performance in video retrieval

Effectively models dynamic and semantically rich content

Robust to noise and redundancies

Abstract

We propose a video feature representation learning framework called STAR-GNN, which applies a pluggable graph neural network component on a multi-scale lattice feature graph. The essence of STAR-GNN is to exploit both the temporal dynamics and spatial contents as well as visual connections between regions at different scales in the frames. It models a video with a lattice feature graph in which the nodes represent regions of different granularity, with weighted edges that represent the spatial and temporal links. The contextual nodes are aggregated simultaneously by graph neural networks with parameters trained with retrieval triplet loss. In the experiments, we show that STAR-GNN effectively implements a dynamic attention mechanism on video frame sequences, resulting in the emphasis for dynamic and semantically rich content in the video, and is robust to noise and redundancies. Empirical results show that STAR-GNN achieves state-of-the-art performance for Content-Based Video Retrieval.