CC-FMO: Camera-Conditioned Zero-Shot Single Image to 3D Scene Generation with Foundation Model Orchestration

TL;DR

CC-FMO is a zero-shot, camera-conditioned pipeline that generates high-quality, coherent 3D scenes from a single image, advancing scene generation by integrating semantics-aware object modeling and scene-level pose estimation.

Contribution

It introduces a novel zero-shot scene generation method that combines a hybrid object generator with camera-conditioned pose estimation for improved scene coherence.

Findings

Outperforms state-of-the-art methods in scene quality and coherence.

Generates high-fidelity, camera-aligned 3D scenes from single images.

Effectively enforces scene-level coherence through camera-conditioned pose estimation.

Abstract

High-quality 3D scene generation from a single image is crucial for AR/VR and embodied AI applications. Early approaches struggle to generalize due to reliance on specialized models trained on curated small datasets. While recent advancements in large-scale 3D foundation models have significantly enhanced instance-level generation, coherent scene generation remains a challenge, where performance is limited by inaccurate per-object pose estimations and spatial inconsistency. To this end, this paper introduces CC-FMO, a zero-shot, camera-conditioned pipeline for single-image to 3D scene generation that jointly conforms to the object layout in input image and preserves instance fidelity. CC-FMO employs a hybrid instance generator that combines semantics-aware vector-set representation with detail-rich structured latent representation, yielding object geometries that are both semantically plausible and high-quality. Furthermore, CC-FMO enables the application of foundational pose estimation models in the scene generation task via a simple yet effective camera-conditioned scale-solving algorithm, to enforce scene-level coherence. Extensive experiments demonstrate that CC-FMO consistently generates high-fidelity camera-aligned compositional scenes, outperforming all state-of-the-art methods.