Pippo: High-Resolution Multi-View Humans from a Single Image

TL;DR

Pippo is a novel generative model that creates high-resolution, multi-view videos of humans from a single image without extra input, using a multi-stage training process and attention biasing for enhanced view synthesis.

Contribution

It introduces a multi-view diffusion transformer trained on large datasets, with novel training and inference techniques for 3D-consistent, high-resolution human generation from a single image.

Findings

Outperforms existing methods in multi-view human generation.

Generates over 5 times more views than during training.

Achieves 1K resolution dense turnaround videos from a single photo.

Abstract

We present Pippo, a generative model capable of producing 1K resolution dense turnaround videos of a person from a single casually clicked photo. Pippo is a multi-view diffusion transformer and does not require any additional inputs - e.g., a fitted parametric model or camera parameters of the input image. We pre-train Pippo on 3B human images without captions, and conduct multi-view mid-training and post-training on studio captured humans. During mid-training, to quickly absorb the studio dataset, we denoise several (up to 48) views at low-resolution, and encode target cameras coarsely using a shallow MLP. During post-training, we denoise fewer views at high-resolution and use pixel-aligned controls (e.g., Spatial anchor and Plucker rays) to enable 3D consistent generations. At inference, we propose an attention biasing technique that allows Pippo to simultaneously generate greater than 5 times as many views as seen during training. Finally, we also introduce an improved metric to evaluate 3D consistency of multi-view generations, and show that Pippo outperforms existing works on multi-view human generation from a single image.