Sparse Pose Trajectory Completion

TL;DR

This paper introduces a novel method for synthesizing dense pose trajectories of objects from sparse view datasets by leveraging cross-modal transfer between RGB images and depth maps, enabling improved novel view synthesis.

Contribution

It presents a cross-modal pose transfer mechanism that generates dense pose trajectories from sparse data, using depth prediction and latent space mapping, which is a new approach in sparse view synthesis.

Findings

Significant improvement in novel view synthesis on Pix3D and ShapeNet datasets.

Effective transfer of pose trajectories to unseen object instances.

Outperforms recent state-of-the-art methods in sparse pose supervision settings.

Abstract

We propose a method to learn, even using a dataset where objects appear only in sparsely sampled views (e.g. Pix3D), the ability to synthesize a pose trajectory for an arbitrary reference image. This is achieved with a cross-modal pose trajectory transfer mechanism. First, a domain transfer function is trained to predict, from an RGB image of the object, its 2D depth map. Then, a set of image views is generated by learning to simulate object rotation in the depth space. Finally, the generated poses are mapped from this latent space into a set of corresponding RGB images using a learned identity preserving transform. This results in a dense pose trajectory of the object in image space. For each object type (e.g., a specific Ikea chair model), a 3D CAD model is used to render a full pose trajectory of 2D depth maps. In the absence of dense pose sampling in image space, these latent space trajectories provide cross-modal guidance for learning. The learned pose trajectories can be transferred to unseen examples, effectively synthesizing all object views in image space. Our method is evaluated on the Pix3D and ShapeNet datasets, in the setting of novel view synthesis under sparse pose supervision, demonstrating substantial improvements over recent art.