Exploring Dual-task Correlation for Pose Guided Person Image Generation

TL;DR

This paper introduces a dual-task pose transformer network that leverages source-to-source and source-to-target tasks with a correlation module to improve person image generation, achieving better detail and efficiency.

Contribution

The novel DPTN uses dual-task learning with a pose transformer module to enhance texture transfer and detail in pose-guided person image generation.

Findings

Outperforms state-of-the-art in PSNR and LPIPS metrics.

Uses significantly fewer parameters than existing methods.

Effectively captures pixel-level correlations for detailed texture transfer.

Abstract

Pose Guided Person Image Generation (PGPIG) is the task of transforming a person image from the source pose to a given target pose. Most of the existing methods only focus on the ill-posed source-to-target task and fail to capture reasonable texture mapping. To address this problem, we propose a novel Dual-task Pose Transformer Network (DPTN), which introduces an auxiliary task (i.e., source-to-source task) and exploits the dual-task correlation to promote the performance of PGPIG. The DPTN is of a Siamese structure, containing a source-to-source self-reconstruction branch, and a transformation branch for source-to-target generation. By sharing partial weights between them, the knowledge learned by the source-to-source task can effectively assist the source-to-target learning. Furthermore, we bridge the two branches with a proposed Pose Transformer Module (PTM) to adaptively explore the correlation between features from dual tasks. Such correlation can establish the fine-grained mapping of all the pixels between the sources and the targets, and promote the source texture transmission to enhance the details of the generated target images. Extensive experiments show that our DPTN outperforms state-of-the-arts in terms of both PSNR and LPIPS. In addition, our DPTN only contains 9.79 million parameters, which is significantly smaller than other approaches. Our code is available at: https://github.com/PangzeCheung/Dual-task-Pose-Transformer-Network.