SOEDiff: Efficient Distillation for Small Object Editing

TL;DR

This paper introduces SOEDiff, a training-based method that enhances small object editing in images by fine-tuning diffusion models with minimal training costs, significantly improving quality and accuracy.

Contribution

We propose SOEDiff, a novel approach combining SO-LoRA and Cross-Scale Score Distillation to improve small object editing in diffusion models with efficient training.

Findings

Improved CLIP-Score by 0.99 on OpenImage-f dataset.

Reduced FID by 2.87, indicating higher image quality.

Validated effectiveness on MSCOCO and OpenImage datasets.

Abstract

In this paper, we delve into a new task known as small object editing (SOE), which focuses on text-based image inpainting within a constrained, small-sized area. Despite the remarkable success have been achieved by current image inpainting approaches, their application to the SOE task generally results in failure cases such as Object Missing, Text-Image Mismatch, and Distortion. These failures stem from the limited use of small-sized objects in training datasets and the downsampling operations employed by U-Net models, which hinders accurate generation. To overcome these challenges, we introduce a novel training-based approach, SOEDiff, aimed at enhancing the capability of baseline models like StableDiffusion in editing small-sized objects while minimizing training costs. Specifically, our method involves two key components: SO-LoRA, which efficiently fine-tunes low-rank matrices, and Cross-Scale Score Distillation loss, which leverages high-resolution predictions from the pre-trained teacher diffusion model. Our method presents significant improvements on the test dataset collected from MSCOCO and OpenImage, validating the effectiveness of our proposed method in small object editing. In particular, when comparing SOEDiff with SD-I model on the OpenImage-f dataset, we observe a 0.99 improvement in CLIP-Score and a reduction of 2.87 in FID.