GUNet: A Graph Convolutional Network United Diffusion Model for Stable and Diversity Pose Generation

TL;DR

This paper introduces GUNet and PoseDiffusion, a novel diffusion-based framework for generating diverse, accurate, and aesthetically pleasing human pose skeletons from textual descriptions, outperforming existing GAN-based methods.

Contribution

It presents the first diffusion model-based framework for text-driven pose skeleton generation, incorporating graph convolutional networks for improved structural accuracy and diversity.

Findings

Outperforms state-of-the-art GAN-based methods in stability and diversity

Incorporates skeletal information via graph convolutional networks for better structural learning

Demonstrates superior controllability and aesthetic quality in pose generation

Abstract

Pose skeleton images are an important reference in pose-controllable image generation. In order to enrich the source of skeleton images, recent works have investigated the generation of pose skeletons based on natural language. These methods are based on GANs. However, it remains challenging to perform diverse, structurally correct and aesthetically pleasing human pose skeleton generation with various textual inputs. To address this problem, we propose a framework with GUNet as the main model, PoseDiffusion. It is the first generative framework based on a diffusion model and also contains a series of variants fine-tuned based on a stable diffusion model. PoseDiffusion demonstrates several desired properties that outperform existing methods. 1) Correct Skeletons. GUNet, a denoising model of PoseDiffusion, is designed to incorporate graphical convolutional neural networks. It is able to learn the spatial relationships of the human skeleton by introducing skeletal information during the training process. 2) Diversity. We decouple the key points of the skeleton and characterise them separately, and use cross-attention to introduce textual conditions. Experimental results show that PoseDiffusion outperforms existing SoTA algorithms in terms of stability and diversity of text-driven pose skeleton generation. Qualitative analyses further demonstrate its superiority for controllable generation in Stable Diffusion.