There and Back Again: On the relation between Noise and Image Inversions in Diffusion Models

TL;DR

This paper analyzes the relationship between noise and image inversions in diffusion models, revealing structural patterns in latents and proposing a simple fix to improve image editing and interpolation capabilities.

Contribution

It introduces a novel method replacing initial DDIM inversion steps with forward diffusion to enhance latent decorrelation and editing quality.

Findings

Latents show less diverse noise in smooth image areas.

First inversion steps are critical for accurate noise representation.

Replacing initial inversion steps improves editing and interpolation.

Abstract

Diffusion Models achieve state-of-the-art performance in generating new samples but lack a low-dimensional latent space that encodes the data into editable features. Inversion-based methods address this by reversing the denoising trajectory, transferring images to their approximated starting noise. In this work, we thoroughly analyze this procedure and focus on the relation between the initial noise, the generated samples, and their corresponding latent encodings obtained through the DDIM inversion. First, we show that latents exhibit structural patterns in the form of less diverse noise predicted for smooth image areas (e.g., plain sky). Through a series of analyses, we trace this issue to the first inversion steps, which fail to provide accurate and diverse noise. Consequently, the DDIM inversion space is notably less manipulative than the original noise. We show that prior inversion methods do not fully resolve this issue, but our simple fix, where we replace the first DDIM Inversion steps with a forward diffusion process, successfully decorrelates latent encodings and enables higher quality editions and interpolations. The code is available at https://github.com/luk-st/taba.