STUPD: A Synthetic Dataset for Spatial and Temporal Relation Reasoning

TL;DR

STUPD is a large-scale synthetic video dataset designed to improve models' understanding of static and dynamic spatial and temporal relations between objects, aiding in bridging visual and language understanding.

Contribution

The paper introduces STUPD, a novel synthetic dataset with 150K videos and images capturing spatial and temporal relations, including 3D object info, to enhance visual relationship reasoning.

Findings

Models pretrained on STUPD outperform others on real-world datasets.

STUPD covers 30 spatial and 10 temporal relation types.

Synthetic data improves spatial and temporal reasoning in vision models.

Abstract

Understanding relations between objects is crucial for understanding the semantics of a visual scene. It is also an essential step in order to bridge visual and language models. However, current state-of-the-art computer vision models still lack the ability to perform spatial reasoning well. Existing datasets mostly cover a relatively small number of spatial relations, all of which are static relations that do not intrinsically involve motion. In this paper, we propose the Spatial and Temporal Understanding of Prepositions Dataset (STUPD) -- a large-scale video dataset for understanding static and dynamic spatial relationships derived from prepositions of the English language. The dataset contains 150K visual depictions (videos and images), consisting of 30 distinct spatial prepositional senses, in the form of object interaction simulations generated synthetically using Unity3D. In addition to spatial relations, we also propose 50K visual depictions across 10 temporal relations, consisting of videos depicting event/time-point interactions. To our knowledge, no dataset exists that represents temporal relations through visual settings. In this dataset, we also provide 3D information about object interactions such as frame-wise coordinates, and descriptions of the objects used. The goal of this synthetic dataset is to help models perform better in visual relationship detection in real-world settings. We demonstrate an increase in the performance of various models over 2 real-world datasets (ImageNet-VidVRD and Spatial Senses) when pretrained on the STUPD dataset, in comparison to other pretraining datasets.