CoPa-SG: Dense Scene Graphs with Parametric and Proto-Relations

TL;DR

This paper introduces CoPa-SG, a synthetic dataset with precise annotations and novel relation types for scene graphs, enhancing scene understanding and reasoning in computer vision.

Contribution

The paper presents a new synthetic dataset and the concepts of parametric and proto-relations, advancing scene graph representation and generation.

Findings

CoPa-SG provides highly accurate ground truth data.

Parametric and proto-relations improve scene graph expressiveness.

Enhanced downstream reasoning capabilities demonstrated.

Abstract

2D scene graphs provide a structural and explainable framework for scene understanding. However, current work still struggles with the lack of accurate scene graph data. To overcome this data bottleneck, we present CoPa-SG, a synthetic scene graph dataset with highly precise ground truth and exhaustive relation annotations between all objects. Moreover, we introduce parametric and proto-relations, two new fundamental concepts for scene graphs. The former provides a much more fine-grained representation than its traditional counterpart by enriching relations with additional parameters such as angles or distances. The latter encodes hypothetical relations in a scene graph and describes how relations would form if new objects are placed in the scene. Using CoPa-SG, we compare the performance of various scene graph generation models. We demonstrate how our new relation types can be integrated in downstream applications to enhance planning and reasoning capabilities.