DiffUHaul: A Training-Free Method for Object Dragging in Images

TL;DR

DiffUHaul is a training-free, diffusion-based method that enables seamless object dragging in images by leveraging spatial reasoning and attention mechanisms, improving real-world image editing without retraining models.

Contribution

The paper introduces DiffUHaul, a novel training-free approach that uses attention masking, diffusion anchoring, and self-attention bucketing for effective object dragging in images.

Findings

Achieves reliable object dragging in real images.

Outperforms existing methods in editing quality.

Validated through user preference studies.

Abstract

Text-to-image diffusion models have proven effective for solving many image editing tasks. However, the seemingly straightforward task of seamlessly relocating objects within a scene remains surprisingly challenging. Existing methods addressing this problem often struggle to function reliably in real-world scenarios due to lacking spatial reasoning. In this work, we propose a training-free method, dubbed DiffUHaul, that harnesses the spatial understanding of a localized text-to-image model, for the object dragging task. Blindly manipulating layout inputs of the localized model tends to cause low editing performance due to the intrinsic entanglement of object representation in the model. To this end, we first apply attention masking in each denoising step to make the generation more disentangled across different objects and adopt the self-attention sharing mechanism to preserve the high-level object appearance. Furthermore, we propose a new diffusion anchoring technique: in the early denoising steps, we interpolate the attention features between source and target images to smoothly fuse new layouts with the original appearance; in the later denoising steps, we pass the localized features from the source images to the interpolated images to retain fine-grained object details. To adapt DiffUHaul to real-image editing, we apply a DDPM self-attention bucketing that can better reconstruct real images with the localized model. Finally, we introduce an automated evaluation pipeline for this task and showcase the efficacy of our method. Our results are reinforced through a user preference study.