HandCraft: Anatomically Correct Restoration of Malformed Hands in Diffusion Generated Images

TL;DR

HandCraft is a novel method that automatically restores anatomically correct human hands in diffusion-generated images by using masks and depth images, without fine-tuning the models, and is evaluated on a new dataset.

Contribution

This paper introduces HandCraft, a plug-and-play approach for correcting malformed hands in diffusion images using parametric models, without requiring model retraining.

Findings

Restores anatomically correct hands in generated images.

Maintains overall image style and pose after correction.

Works with existing pretrained diffusion models.

Abstract

Generative text-to-image models, such as Stable Diffusion, have demonstrated a remarkable ability to generate diverse, high-quality images. However, they are surprisingly inept when it comes to rendering human hands, which are often anatomically incorrect or reside in the "uncanny valley". In this paper, we propose a method HandCraft for restoring such malformed hands. This is achieved by automatically constructing masks and depth images for hands as conditioning signals using a parametric model, allowing a diffusion-based image editor to fix the hand's anatomy and adjust its pose while seamlessly integrating the changes into the original image, preserving pose, color, and style. Our plug-and-play hand restoration solution is compatible with existing pretrained diffusion models, and the restoration process facilitates adoption by eschewing any fine-tuning or training requirements for the diffusion models. We also contribute MalHand datasets that contain generated images with a wide variety of malformed hands in several styles for hand detector training and hand restoration benchmarking, and demonstrate through qualitative and quantitative evaluation that HandCraft not only restores anatomical correctness but also maintains the integrity of the overall image.