Unsupervised Volumetric Animation

TL;DR

This paper introduces an unsupervised method for 3D animation of deformable objects from single-view videos, enabling segmentation, keypoint estimation, and novel view synthesis without labeled data.

Contribution

It presents a novel 3D autodecoder framework combined with a differentiable PnP for unsupervised learning of object geometry and parts from videos and images.

Findings

Effective 3D segmentation and keypoint estimation from videos.

Capable of novel view synthesis and animation.

Learns 3D geometry from still images.

Abstract

We propose a novel approach for unsupervised 3D animation of non-rigid deformable objects. Our method learns the 3D structure and dynamics of objects solely from single-view RGB videos, and can decompose them into semantically meaningful parts that can be tracked and animated. Using a 3D autodecoder framework, paired with a keypoint estimator via a differentiable PnP algorithm, our model learns the underlying object geometry and parts decomposition in an entirely unsupervised manner. This allows it to perform 3D segmentation, 3D keypoint estimation, novel view synthesis, and animation. We primarily evaluate the framework on two video datasets: VoxCeleb $256^2$ and TEDXPeople $256^2$. In addition, on the Cats $256^2$ image dataset, we show it even learns compelling 3D geometry from still images. Finally, we show our model can obtain animatable 3D objects from a single or few images. Code and visual results available on our project website, see https://snap-research.github.io/unsupervised-volumetric-animation .