RHanDS: Refining Malformed Hands for Generated Images with Decoupled Structure and Style Guidance

TL;DR

RHanDS is a diffusion-based framework that improves the structural accuracy of generated hands while maintaining style, using decoupled guidance and a two-stage training process on multi-style datasets.

Contribution

The paper introduces a novel two-stage training strategy and decoupled guidance for refining malformed hands in generated images, enhancing structural correctness without losing style.

Findings

Effective hand structure refinement demonstrated in experiments

Preserves hand style while correcting structure

Utilizes multi-style datasets for robust training

Abstract

Although diffusion models can generate high-quality human images, their applications are limited by the instability in generating hands with correct structures. In this paper, we introduce RHanDS, a conditional diffusion-based framework designed to refine malformed hands by utilizing decoupled structure and style guidance. The hand mesh reconstructed from the malformed hand offers structure guidance for correcting the structure of the hand, while the malformed hand itself provides style guidance for preserving the style of the hand. To alleviate the mutual interference between style and structure guidance, we introduce a two-stage training strategy and build a series of multi-style hand datasets. In the first stage, we use paired hand images for training to ensure stylistic consistency in hand refining. In the second stage, various hand images generated based on human meshes are used for training, enabling the model to gain control over the hand structure. Experimental results demonstrate that RHanDS can effectively refine hand structure while preserving consistency in hand style.