TabletopGen: Instance-Level Interactive 3D Tabletop Scene Generation from Text or Single Image

TL;DR

TabletopGen is a training-free framework that generates diverse, high-fidelity, physically interactive 3D tabletop scenes from a reference image, improving scene realism and diversity for embodied AI applications.

Contribution

It introduces a novel pose and scale alignment method and a fully automatic pipeline for instance-level 3D scene generation from images or text.

Findings

Achieves state-of-the-art visual fidelity and layout accuracy.

Surpasses existing methods in physical plausibility and diversity.

Enables realistic and diverse tabletop scene generation.

Abstract

Generating high-fidelity, physically interactive 3D simulated tabletop scenes is essential for embodied AI -- especially for robotic manipulation policy learning and data synthesis. However, current text- or image-driven 3D scene generation methods mainly focus on large-scale scenes, struggling to capture the high-density layouts and complex spatial relations that characterize tabletop scenes. To address these challenges, we propose TabletopGen, a training-free, fully automatic framework that generates diverse, instance-level interactive 3D tabletop scenes. TabletopGen accepts a reference image as input, which can be synthesized by a text-to-image model to enhance scene diversity. We then perform instance segmentation and completion on the reference to obtain per-instance images. Each instance is reconstructed into a 3D model followed by canonical coordinate alignment. The aligned 3D models then undergo pose and scale estimation before being assembled into a collision-free, simulation-ready tabletop scene. A key component of our framework is a novel pose and scale alignment approach that decouples the complex spatial reasoning into two stages: a Differentiable Rotation Optimizer for precise rotation recovery and a Top-view Spatial Alignment mechanism for robust translation and scale estimation, enabling accurate 3D reconstruction from 2D reference. Extensive experiments and user studies show that TabletopGen achieves state-of-the-art performance, markedly surpassing existing methods in visual fidelity, layout accuracy, and physical plausibility, capable of generating realistic tabletop scenes with rich stylistic and spatial diversity. Our code will be publicly available.