CLIPDrag: Combining Text-based and Drag-based Instructions for Image Editing

TL;DR

CLIPDrag is a novel image editing approach that combines text and drag signals to achieve precise, flexible, and ambiguity-free modifications, leveraging CLIP and a global-local supervision strategy.

Contribution

This paper introduces CLIPDrag, the first method to integrate text and drag instructions for image editing, improving accuracy and convergence speed.

Findings

Outperforms existing drag-based and text-based methods.

Achieves more precise and unambiguous image edits.

Speeds up convergence with a new point-tracking technique.

Abstract

Precise and flexible image editing remains a fundamental challenge in computer vision. Based on the modified areas, most editing methods can be divided into two main types: global editing and local editing. In this paper, we choose the two most common editing approaches (ie text-based editing and drag-based editing) and analyze their drawbacks. Specifically, text-based methods often fail to describe the desired modifications precisely, while drag-based methods suffer from ambiguity. To address these issues, we proposed \textbf{CLIPDrag}, a novel image editing method that is the first to combine text and drag signals for precise and ambiguity-free manipulations on diffusion models. To fully leverage these two signals, we treat text signals as global guidance and drag points as local information. Then we introduce a novel global-local motion supervision method to integrate text signals into existing drag-based methods by adapting a pre-trained language-vision model like CLIP. Furthermore, we also address the problem of slow convergence in CLIPDrag by presenting a fast point-tracking method that enforces drag points moving toward correct directions. Extensive experiments demonstrate that CLIPDrag outperforms existing single drag-based methods or text-based methods.