DragLoRA: Online Optimization of LoRA Adapters for Drag-based Image Editing in Diffusion Model

TL;DR

DragLoRA introduces a novel framework integrating LoRA adapters into diffusion model editing, significantly improving control precision and efficiency in drag-based image manipulation.

Contribution

It presents a new online optimization method for LoRA adapters, with a regularization loss and adaptive scheme, enhancing accuracy and efficiency in diffusion model editing.

Findings

Enhanced control precision in drag-based editing

Improved computational efficiency

Effective online optimization of LoRA adapters

Abstract

Drag-based editing within pretrained diffusion model provides a precise and flexible way to manipulate foreground objects. Traditional methods optimize the input feature obtained from DDIM inversion directly, adjusting them iteratively to guide handle points towards target locations. However, these approaches often suffer from limited accuracy due to the low representation ability of the feature in motion supervision, as well as inefficiencies caused by the large search space required for point tracking. To address these limitations, we present DragLoRA, a novel framework that integrates LoRA (Low-Rank Adaptation) adapters into the drag-based editing pipeline. To enhance the training of LoRA adapters, we introduce an additional denoising score distillation loss which regularizes the online model by aligning its output with that of the original model. Additionally, we improve the consistency of motion supervision by adapting the input features using the updated LoRA, giving a more stable and accurate input feature for subsequent operations. Building on this, we design an adaptive optimization scheme that dynamically toggles between two modes, prioritizing efficiency without compromising precision. Extensive experiments demonstrate that DragLoRA significantly enhances the control precision and computational efficiency for drag-based image editing. The Codes of DragLoRA are available at: https://github.com/Sylvie-X/DragLoRA.