From Vision To Language through Graph of Events in Space and Time: An Explainable Self-supervised Approach

TL;DR

This paper introduces an explainable, graph-based representation of events in space and time to improve video captioning, enabling better understanding and self-supervised training of models for generating detailed natural language descriptions.

Contribution

It proposes a novel explainable graph-based approach for connecting vision and language, and demonstrates its effectiveness as a self-supervised teacher for training video captioning models.

Findings

Generated coherent and relevant descriptions across multiple datasets.

Validated approach with standard metrics, human annotations, and ensemble consensus.

Enabled self-supervised training of neural models using the explainable system.

Abstract

The task of describing video content in natural language is commonly referred to as video captioning. Unlike conventional video captions, which are typically brief and widely available, long-form paragraph descriptions in natural language are scarce. This limitation of current datasets is due to the expensive human manual annotation required and to the highly challenging task of explaining the language formation process from the perspective of the underlying story, as a complex system of interconnected events in space and time. Through a thorough analysis of recently published methods and available datasets, we identify a general lack of published resources dedicated to the problem of describing videos in complex language, beyond the level of descriptions in the form of enumerations of simple captions. Furthermore, while state-of-the-art methods produce impressive results on the task of generating shorter captions from videos by direct end-to-end learning between the videos and text, the problem of explaining the relationship between vision and language is still beyond our reach. In this work, we propose a shared representation between vision and language, based on graphs of events in space and time, which can be obtained in an explainable and analytical way, to integrate and connect multiple vision tasks to produce the final natural language description. Moreover, we also demonstrate how our automated and explainable video description generation process can function as a fully automatic teacher to effectively train direct, end-to-end neural student pathways, within a self-supervised neuro-analytical system. We validate that our explainable neuro-analytical approach generates coherent, rich and relevant textual descriptions on videos collected from multiple varied datasets, using both standard evaluation metrics, human annotations and consensus from ensembles of state-of-the-art VLMs.