Geometry-Guided Reinforcement Learning for Multi-view Consistent 3D Scene Editing

TL;DR

This paper introduces RL3DEdit, a reinforcement learning framework that uses 3D priors from a foundation model to achieve multi-view consistent 3D scene editing without requiring extensive 3D-consistent paired data.

Contribution

It proposes a novel RL-based approach leveraging 3D priors for consistent 3D editing, addressing the challenge of multi-view consistency without supervised fine-tuning.

Findings

Achieves stable multi-view consistency in 3D editing.

Outperforms state-of-the-art methods in editing quality.

Demonstrates high efficiency in 3D scene editing.

Abstract

Leveraging the priors of 2D diffusion models for 3D editing has emerged as a promising paradigm. However, maintaining multi-view consistency in edited results remains challenging, and the extreme scarcity of 3D-consistent editing paired data renders supervised fine-tuning (SFT), the most effective training strategy for editing tasks, infeasible. In this paper, we observe that, while generating multi-view consistent 3D content is highly challenging, verifying 3D consistency is tractable, naturally positioning reinforcement learning (RL) as a feasible solution. Motivated by this, we propose \textbf{RL3DEdit}, a single-pass framework driven by RL optimization with novel rewards derived from the 3D foundation model, VGGT. Specifically, we leverage VGGT's robust priors learned from massive real-world data, feed the edited images, and utilize the output confidence maps and pose estimation errors as reward signals, effectively anchoring the 2D editing priors onto a 3D-consistent manifold via RL. Extensive experiments demonstrate that RL3DEdit achieves stable multi-view consistency and outperforms state-of-the-art methods in editing quality with high efficiency. To promote the development of 3D editing, we will release the code and model.