SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentation

TL;DR

SceneGraphNet introduces a neural message passing framework that predicts and augments 3D indoor scenes with contextually fitting objects, outperforming existing methods in object prediction accuracy.

Contribution

The paper presents a novel neural message passing approach with attention mechanisms for scene augmentation, improving object prediction in 3D indoor environments.

Findings

Outperforms state-of-the-art in object prediction accuracy

Effective in context-based 3D object recognition

Enables iterative scene generation

Abstract

In this paper we propose a neural message passing approach to augment an input 3D indoor scene with new objects matching their surroundings. Given an input, potentially incomplete, 3D scene and a query location, our method predicts a probability distribution over object types that fit well in that location. Our distribution is predicted though passing learned messages in a dense graph whose nodes represent objects in the input scene and edges represent spatial and structural relationships. By weighting messages through an attention mechanism, our method learns to focus on the most relevant surrounding scene context to predict new scene objects. We found that our method significantly outperforms state-of-the-art approaches in terms of correctly predicting objects missing in a scene based on our experiments in the SUNCG dataset. We also demonstrate other applications of our method, including context-based 3D object recognition and iterative scene generation.