ArtiFade: Learning to Generate High-quality Subject from Blemished Images

TL;DR

ArtiFade is a novel method that fine-tunes pre-trained text-to-image models to effectively remove artifacts from blemished images, enabling high-quality, subject-specific image generation even with imperfect input data.

Contribution

It introduces a fine-tuning approach using a specialized dataset to eliminate artifacts while preserving the model's generative capabilities, improving artifact removal in subject-driven image synthesis.

Findings

Effective artifact removal in blemished images

Maintains high-quality subject generation

Works in both in-distribution and out-of-distribution scenarios

Abstract

Subject-driven text-to-image generation has witnessed remarkable advancements in its ability to learn and capture characteristics of a subject using only a limited number of images. However, existing methods commonly rely on high-quality images for training and may struggle to generate reasonable images when the input images are blemished by artifacts. This is primarily attributed to the inadequate capability of current techniques in distinguishing subject-related features from disruptive artifacts. In this paper, we introduce ArtiFade to tackle this issue and successfully generate high-quality artifact-free images from blemished datasets. Specifically, ArtiFade exploits fine-tuning of a pre-trained text-to-image model, aiming to remove artifacts. The elimination of artifacts is achieved by utilizing a specialized dataset that encompasses both unblemished images and their corresponding blemished counterparts during fine-tuning. ArtiFade also ensures the preservation of the original generative capabilities inherent within the diffusion model, thereby enhancing the overall performance of subject-driven methods in generating high-quality and artifact-free images. We further devise evaluation benchmarks tailored for this task. Through extensive qualitative and quantitative experiments, we demonstrate the generalizability of ArtiFade in effective artifact removal under both in-distribution and out-of-distribution scenarios.