Weak Multi-View Supervision for Surface Mapping Estimation

TL;DR

This paper introduces a weakly-supervised multi-view learning method for surface mapping that leverages cycle consistency and deformation fields, reducing the need for dense annotations and achieving results comparable to fully supervised approaches.

Contribution

It presents a novel multi-view weak supervision framework using cycle consistency and deformation fields for surface mapping without dense annotations.

Findings

Achieves multi-view consistent surface mapping comparable to supervised methods.

Introduces a new multi-view dataset for cars and airplanes.

Demonstrates accurate shape variations away from the mean shape.

Abstract

We propose a weakly-supervised multi-view learning approach to learn category-specific surface mapping without dense annotations. We learn the underlying surface geometry of common categories, such as human faces, cars, and airplanes, given instances from those categories. While traditional approaches solve this problem using extensive supervision in the form of pixel-level annotations, we take advantage of the fact that pixel-level UV and mesh predictions can be combined with 3D reprojections to form consistency cycles. As a result of exploiting these cycles, we can establish a dense correspondence mapping between image pixels and the mesh acting as a self-supervisory signal, which in turn helps improve our overall estimates. Our approach leverages information from multiple views of the object to establish additional consistency cycles, thus improving surface mapping understanding without the need for explicit annotations. We also propose the use of deformation fields for predictions of an instance specific mesh. Given the lack of datasets providing multiple images of similar object instances from different viewpoints, we generate and release a multi-view ShapeNet Cars and Airplanes dataset created by rendering ShapeNet meshes using a 360 degree camera trajectory around the mesh. For the human faces category, we process and adapt an existing dataset to a multi-view setup. Through experimental evaluations, we show that, at test time, our method can generate accurate variations away from the mean shape, is multi-view consistent, and performs comparably to fully supervised approaches.