Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment

TL;DR

This paper introduces Finsta, a novel fine-grained structural spatio-temporal alignment method that enhances video-language models by representing data with scene graphs and employing graph Transformers for improved cross-modal understanding.

Contribution

Finsta is a plug-and-play framework that unifies scene graph representations for texts and videos, improving alignment and performance without retraining from scratch or needing additional annotations.

Findings

Consistently improves 13 strong VLMs across 12 datasets.

Significantly enhances performance in both fine-tuning and zero-shot settings.

Achieves state-of-the-art results on multiple video-language tasks.

Abstract

While pre-training large-scale video-language models (VLMs) has shown remarkable potential for various downstream video-language tasks, existing VLMs can still suffer from certain commonly seen limitations, e.g., coarse-grained cross-modal aligning , under-modeling of temporal dynamics, detached video-language view. In this work, we target enhancing VLMs with a fine-grained structural spatio-temporal alignment learning method (namely Finsta). First of all, we represent the input texts and videos with fine-grained scene graph (SG) structures, both of which are further unified into a holistic SG (HSG) for bridging two modalities. Then, an SG-based framework is built, where the textual SG (TSG) is encoded with a graph Transformer, while the video dynamic SG (DSG) and the HSG are modeled with a novel recurrent graph Transformer for spatial and temporal feature propagation. A spatial-temporal Gaussian differential graph Transformer is further devised to strengthen the sense of the changes in objects across spatial and temporal dimensions. Next, based on the fine-grained structural features of TSG and DSG, we perform object-centered spatial alignment and predicate-centered temporal alignment respectively, enhancing the video-language grounding in both the spatiality and temporality. We design our method as a plug&play system, which can be integrated into existing well-trained VLMs for further representation augmentation, without training from scratch or relying on SG annotations in downstream applications. On 6 representative VL modeling tasks over 12 datasets in both standard and long-form video scenarios, Finsta consistently improves the existing 13 strong-performing VLMs persistently, and refreshes the current state-of-the-art end task performance significantly in both the fine-tuning and zero-shot settings.