Explaining Vision and Language through Graphs of Events in Space and Time

TL;DR

This paper introduces GEST, a graph-based model that represents and explains visual and linguistic stories in space and time, enhancing video generation and semantic comparison.

Contribution

The paper proposes GEST, a novel graph-based framework that bridges vision and language, providing explainability and improving video generation and text comparison.

Findings

GEST improves video content generation from text.

GEST enhances semantic comparison between texts.

Theoretical and experimental validation supports GEST's effectiveness.

Abstract

Artificial Intelligence makes great advances today and starts to bridge the gap between vision and language. However, we are still far from understanding, explaining and controlling explicitly the visual content from a linguistic perspective, because we still lack a common explainable representation between the two domains. In this work we come to address this limitation and propose the Graph of Events in Space and Time (GEST), by which we can represent, create and explain, both visual and linguistic stories. We provide a theoretical justification of our model and an experimental validation, which proves that GEST can bring a solid complementary value along powerful deep learning models. In particular, GEST can help improve at the content-level the generation of videos from text, by being easily incorporated into our novel video generation engine. Additionally, by using efficient graph matching techniques, the GEST graphs can also improve the comparisons between texts at the semantic level.