SceneLinker: Compositional 3D Scene Generation via Semantic Scene Graph from RGB Sequences

TL;DR

SceneLinker is a new framework that generates realistic 3D scenes from RGB sequences using semantic scene graphs, improving the alignment of virtual content with real-world object arrangements for mixed reality applications.

Contribution

It introduces a novel graph network with cross-check feature attention and a graph-VAE for accurate 3D scene generation from RGB data, addressing limitations of prior methods.

Findings

Outperforms state-of-the-art methods on 3RScan/3DSSG and SG-FRONT datasets.

Effective in complex indoor environments with scene graph constraints.

Enables consistent 3D space creation from physical environments for MR.

Abstract

We introduce SceneLinker, a novel framework that generates compositional 3D scenes via semantic scene graph from RGB sequences. To adaptively experience Mixed Reality (MR) content based on each user's space, it is essential to generate a 3D scene that reflects the real-world layout by compactly capturing the semantic cues of the surroundings. Prior works struggled to fully capture the contextual relationship between objects or mainly focused on synthesizing diverse shapes, making it challenging to generate 3D scenes aligned with object arrangements. We address these challenges by designing a graph network with cross-check feature attention for scene graph prediction and constructing a graph-variational autoencoder (graph-VAE), which consists of a joint shape and layout block for 3D scene generation. Experiments on the 3RScan/3DSSG and SG-FRONT datasets demonstrate that our approach outperforms state-of-the-art methods in both quantitative and qualitative evaluations, even in complex indoor environments and under challenging scene graph constraints. Our work enables users to generate consistent 3D spaces from their physical environments via scene graphs, allowing them to create spatial MR content. Project page is https://scenelinker2026.github.io.