InpaintSLat: Inpainting Structured 3D Latents via Initial Noise Optimization

TL;DR

This paper introduces a training-free method for controllable 3D inpainting by optimizing initial noise in structured 3D latent diffusion, improving fidelity and contextual consistency.

Contribution

It proposes a novel initial noise optimization strategy using backpropagation approximation, enhancing 3D inpainting stability without additional training.

Findings

Improved contextual consistency in 3D inpainting results.

Enhanced prompt alignment compared to baseline methods.

Demonstrated robustness and efficiency of the proposed optimization approach.

Abstract

We present a training-free approach for controllable 3D inpainting based on initial noise optimization. In the structured 3D latent diffusion framework, we observe that the underlying geometric structure is established during the early stages of the diffusion process and exhibits high sensitivity to the initial noise. Such characteristics compromise stability in tasks like inpainting and editing, where the model must ensure strict alignment with the existing context while synthesizing a new structure. In this paper, we introduce a strategy to optimize the initial noise within the structured 3D latent diffusion framework, ensuring high-fidelity 3D inpainting. Specifically, we update the initial noise by leveraging a backpropagation approximation grounded in the rectified flow model, with the spectral parameterization specially designed for robust and efficient structured 3D latent optimization. Experiments demonstrate consistent improvements in contextual consistency and prompt alignment over representative training-free inpainting baselines, establishing initial noise control as an independent dimension for 3D inpainting, orthogonal to conventional sampling trajectory manipulation.