JieHua Paintings Style Feature Extracting Model using Stable Diffusion with ControlNet

TL;DR

This paper introduces a novel style feature extraction method for Jiehua paintings using a fine-tuned Stable Diffusion Model with ControlNet, outperforming CycleGAN in style transfer quality and preserving semantic content.

Contribution

The study presents a new approach combining Stable Diffusion and ControlNet for Jiehua style extraction, with optimized hyperparameters showing superior performance over existing models.

Findings

FSDMC achieves an FID of 3.27 on Jiehua dataset.

FSDMC surpasses CycleGAN in expert evaluations.

The method effectively preserves semantic information during style transfer.

Abstract

This study proposes a novel approach to extract stylistic features of Jiehua: the utilization of the Fine-tuned Stable Diffusion Model with ControlNet (FSDMC) to refine depiction techniques from artists' Jiehua. The training data for FSDMC is based on the opensource Jiehua artist's work collected from the Internet, which were subsequently manually constructed in the format of (Original Image, Canny Edge Features, Text Prompt). By employing the optimal hyperparameters identified in this paper, it was observed FSDMC outperforms CycleGAN, another mainstream style transfer model. FSDMC achieves FID of 3.27 on the dataset and also surpasses CycleGAN in terms of expert evaluation. This not only demonstrates the model's high effectiveness in extracting Jiehua's style features, but also preserves the original pre-trained semantic information. The findings of this study suggest that the application of FSDMC with appropriate hyperparameters can enhance the efficacy of the Stable Diffusion Model in the field of traditional art style migration tasks, particularly within the context of Jiehua.