SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Model

TL;DR

This paper introduces SynArtifact, a vision-language model fine-tuned to classify and reduce artifacts in synthetic images, significantly enhancing image quality through an artifact-aware feedback loop.

Contribution

It presents the first end-to-end artifact classification method using a fine-tuned VLM and demonstrates its effectiveness in improving synthetic image quality.

Findings

VLM outperforms baseline by 25.66% in artifact classification

Constructed SynArtifact-1K dataset for artifact annotation

Refined diffusion models produce visibly better images

Abstract

In the rapidly evolving area of image synthesis, a serious challenge is the presence of complex artifacts that compromise perceptual realism of synthetic images. To alleviate artifacts and improve quality of synthetic images, we fine-tune Vision-Language Model (VLM) as artifact classifier to automatically identify and classify a wide range of artifacts and provide supervision for further optimizing generative models. Specifically, we develop a comprehensive artifact taxonomy and construct a dataset of synthetic images with artifact annotations for fine-tuning VLM, named SynArtifact-1K. The fine-tuned VLM exhibits superior ability of identifying artifacts and outperforms the baseline by 25.66%. To our knowledge, this is the first time such end-to-end artifact classification task and solution have been proposed. Finally, we leverage the output of VLM as feedback to refine the generative model for alleviating artifacts. Visualization results and user study demonstrate that the quality of images synthesized by the refined diffusion model has been obviously improved.