RealisHuman: A Two-Stage Approach for Refining Malformed Human Parts in Generated Images

TL;DR

RealisHuman is a two-stage post-processing framework that refines malformed human parts in generated images, significantly improving realism by generating and seamlessly integrating realistic human features.

Contribution

It introduces a novel two-stage method for refining human parts in generated images, addressing structural complexity challenges in diffusion-based models.

Findings

Enhanced realism of human parts in generated images

Significant improvements in qualitative and quantitative metrics

Effective blending of refined parts with original images

Abstract

In recent years, diffusion models have revolutionized visual generation, outperforming traditional frameworks like Generative Adversarial Networks (GANs). However, generating images of humans with realistic semantic parts, such as hands and faces, remains a significant challenge due to their intricate structural complexity. To address this issue, we propose a novel post-processing solution named RealisHuman. The RealisHuman framework operates in two stages. First, it generates realistic human parts, such as hands or faces, using the original malformed parts as references, ensuring consistent details with the original image. Second, it seamlessly integrates the rectified human parts back into their corresponding positions by repainting the surrounding areas to ensure smooth and realistic blending. The RealisHuman framework significantly enhances the realism of human generation, as demonstrated by notable improvements in both qualitative and quantitative metrics. Code is available at https://github.com/Wangbenzhi/RealisHuman.