BlenderFusion: 3D-Grounded Visual Editing and Generative Compositing

TL;DR

BlenderFusion is a novel framework that enables 3D-grounded visual editing and scene compositing by integrating segmentation, editing, and a generative diffusion-based compositor, allowing flexible and high-quality scene modifications.

Contribution

It introduces a layered editing pipeline combined with a diffusion-based generative compositor fine-tuned for scene editing, which is a new approach in 3D scene compositing.

Findings

Outperforms prior methods in complex scene editing tasks

Enables flexible background replacement and object manipulation

Provides disentangled control over objects and camera movements

Abstract

We present BlenderFusion, a generative visual compositing framework that synthesizes new scenes by recomposing objects, camera, and background. It follows a layering-editing-compositing pipeline: (i) segmenting and converting visual inputs into editable 3D entities (layering), (ii) editing them in Blender with 3D-grounded control (editing), and (iii) fusing them into a coherent scene using a generative compositor (compositing). Our generative compositor extends a pre-trained diffusion model to process both the original (source) and edited (target) scenes in parallel. It is fine-tuned on video frames with two key training strategies: (i) source masking, enabling flexible modifications like background replacement; (ii) simulated object jittering, facilitating disentangled control over objects and camera. BlenderFusion significantly outperforms prior methods in complex compositional scene editing tasks.