POLARIS: Projection-Orthogonal Least Squares for Robust and Adaptive Inversion in Diffusion Models

TL;DR

POLARIS introduces a mathematically grounded method to improve inversion accuracy in diffusion models by adaptively minimizing noise approximation errors, enhancing downstream task performance with minimal code changes.

Contribution

It reformulates inversion as an error-origin problem and derives a formula to adaptively minimize noise errors at each step, improving robustness and accuracy.

Findings

Significantly reduces noise approximation errors in inversion.

Improves downstream task accuracy with minimal code modifications.

Maintains low computational overhead during inversion.

Abstract

The Inversion-Denoising Paradigm, which is based on diffusion models, excels in diverse image editing and restoration tasks. We revisit its mechanism and reveal a critical, overlooked factor in reconstruction degradation: the approximate noise error. This error stems from approximating the noise at step t with the prediction at step t-1, resulting in severe error accumulation throughout the inversion process. We introduce Projection-Orthogonal Least Squares for Robust and Adaptive Inversion (POLARIS), which reformulates inversion from an error-compensation problem into an error-origin problem. Rather than optimizing embeddings or latent codes to offset accumulated drift, POLARIS treats the guidance scale {\omega} as a step-wise variable and derives a mathematically grounded formula to minimize inversion error at each step. Remarkably, POLARIS improves inversion latent quality with just one line of code. With negligible performance overhead, it substantially mitigates noise approximation errors and consistently improves the accuracy of downstream tasks.