Self-supervised Correlation Mining Network for Person Image Generation

TL;DR

This paper introduces SCM-Net, a self-supervised network that leverages spatial correlation in feature space for person image generation, improving realism and structure preservation without requiring paired data.

Contribution

The paper proposes a novel self-supervised correlation mining network with modules for feature rearrangement and body structure retention, advancing person image generation techniques.

Findings

Outperforms supervised and unsupervised methods on DeepFashion dataset

Effectively preserves body structure in full and half-body generation

Demonstrates versatility in face generation tasks

Abstract

Person image generation aims to perform non-rigid deformation on source images, which generally requires unaligned data pairs for training. Recently, self-supervised methods express great prospects in this task by merging the disentangled representations for self-reconstruction. However, such methods fail to exploit the spatial correlation between the disentangled features. In this paper, we propose a Self-supervised Correlation Mining Network (SCM-Net) to rearrange the source images in the feature space, in which two collaborative modules are integrated, Decomposed Style Encoder (DSE) and Correlation Mining Module (CMM). Specifically, the DSE first creates unaligned pairs at the feature level. Then, the CMM establishes the spatial correlation field for feature rearrangement. Eventually, a translation module transforms the rearranged features to realistic results. Meanwhile, for improving the fidelity of cross-scale pose transformation, we propose a graph based Body Structure Retaining Loss (BSR Loss) to preserve reasonable body structures on half body to full body generation. Extensive experiments conducted on DeepFashion dataset demonstrate the superiority of our method compared with other supervised and unsupervised approaches. Furthermore, satisfactory results on face generation show the versatility of our method in other deformation tasks.