Learning Realistic Human Reposing using Cyclic Self-Supervision with 3D Shape, Pose, and Appearance Consistency

TL;DR

This paper introduces SPICE, a self-supervised framework for realistic human reposing from a single image using 3D shape, pose, and appearance consistency, achieving state-of-the-art results without paired data.

Contribution

The paper presents a novel self-supervised method leveraging 3D human body information to generate realistic reposed images, eliminating the need for paired training data.

Findings

Achieves state-of-the-art FID score of 7.8 on DeepFashion dataset.

Generates temporally coherent videos from static images and pose sequences.

Performs comparably to supervised methods despite using only static images for training.

Abstract

Synthesizing images of a person in novel poses from a single image is a highly ambiguous task. Most existing approaches require paired training images; i.e. images of the same person with the same clothing in different poses. However, obtaining sufficiently large datasets with paired data is challenging and costly. Previous methods that forego paired supervision lack realism. We propose a self-supervised framework named SPICE (Self-supervised Person Image CrEation) that closes the image quality gap with supervised methods. The key insight enabling self-supervision is to exploit 3D information about the human body in several ways. First, the 3D body shape must remain unchanged when reposing. Second, representing body pose in 3D enables reasoning about self occlusions. Third, 3D body parts that are visible before and after reposing, should have similar appearance features. Once trained, SPICE takes an image of a person and generates a new image of that person in a new target pose. SPICE achieves state-of-the-art performance on the DeepFashion dataset, improving the FID score from 29.9 to 7.8 compared with previous unsupervised methods, and with performance similar to the state-of-the-art supervised method (6.4). SPICE also generates temporally coherent videos given an input image and a sequence of poses, despite being trained on static images only.