Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs

TL;DR

This paper introduces an end-to-end method for generating and manipulating 3D scenes directly from scene graphs, enabling detailed control and scene modification without relying on synthetic data or mesh retrieval.

Contribution

It is the first to directly generate 3D shapes from scene graphs using an end-to-end approach with GCNs, supporting scene editing via the same interface.

Findings

Supports direct shape generation from scene graphs

Enables scene modification through scene graph interface

Uses GCN-based variational Auto-Encoder for diverse scene sampling

Abstract

Controllable scene synthesis consists of generating 3D information that satisfy underlying specifications. Thereby, these specifications should be abstract, i.e. allowing easy user interaction, whilst providing enough interface for detailed control. Scene graphs are representations of a scene, composed of objects (nodes) and inter-object relationships (edges), proven to be particularly suited for this task, as they allow for semantic control on the generated content. Previous works tackling this task often rely on synthetic data, and retrieve object meshes, which naturally limits the generation capabilities. To circumvent this issue, we instead propose the first work that directly generates shapes from a scene graph in an end-to-end manner. In addition, we show that the same model supports scene modification, using the respective scene graph as interface. Leveraging Graph Convolutional Networks (GCN) we train a variational Auto-Encoder on top of the object and edge categories, as well as 3D shapes and scene layouts, allowing latter sampling of new scenes and shapes.