UltraEdit: Instruction-based Fine-Grained Image Editing at Scale

TL;DR

UltraEdit introduces a large-scale, high-quality dataset for instruction-based image editing, leveraging real images, diverse instructions, and region annotations to improve editing models and benchmarks.

Contribution

It provides a novel, extensive dataset for image editing that combines real images, diverse instructions, and region annotations, addressing limitations of prior datasets.

Findings

Models trained on UltraEdit achieve new state-of-the-art results on benchmarks.

Real image anchors and region-based data significantly improve editing performance.

UltraEdit enables broader and more accurate image editing capabilities.

Abstract

This paper presents UltraEdit, a large-scale (approximately 4 million editing samples), automatically generated dataset for instruction-based image editing. Our key idea is to address the drawbacks in existing image editing datasets like InstructPix2Pix and MagicBrush, and provide a systematic approach to producing massive and high-quality image editing samples. UltraEdit offers several distinct advantages: 1) It features a broader range of editing instructions by leveraging the creativity of large language models (LLMs) alongside in-context editing examples from human raters; 2) Its data sources are based on real images, including photographs and artworks, which provide greater diversity and reduced bias compared to datasets solely generated by text-to-image models; 3) It also supports region-based editing, enhanced by high-quality, automatically produced region annotations. Our experiments show that canonical diffusion-based editing baselines trained on UltraEdit set new records on MagicBrush and Emu-Edit benchmarks. Our analysis further confirms the crucial role of real image anchors and region-based editing data. The dataset, code, and models can be found in https://ultra-editing.github.io.