Specular-to-Diffuse Translation for Multi-View Reconstruction

TL;DR

This paper presents S2Dnet, a generative adversarial network that converts specular object views into diffuse ones, enhancing multi-view 3D reconstruction accuracy without requiring additional scene information.

Contribution

We introduce a novel multi-view coherence loss for unsupervised specular-to-diffuse translation, improving multi-view reconstruction results significantly.

Findings

S2Dnet outperforms single-view baselines in preserving appearance.

Multi-view coherence loss maintains local patch similarity across views.

The method improves reconstruction quality on real-world data.

Abstract

Most multi-view 3D reconstruction algorithms, especially when shape-from-shading cues are used, assume that object appearance is predominantly diffuse. To alleviate this restriction, we introduce S2Dnet, a generative adversarial network for transferring multiple views of objects with specular reflection into diffuse ones, so that multi-view reconstruction methods can be applied more effectively. Our network extends unsupervised image-to-image translation to multi-view "specular to diffuse" translation. To preserve object appearance across multiple views, we introduce a Multi-View Coherence loss (MVC) that evaluates the similarity and faithfulness of local patches after the view-transformation. Our MVC loss ensures that the similarity of local correspondences among multi-view images is preserved under the image-to-image translation. As a result, our network yields significantly better results than several single-view baseline techniques. In addition, we carefully design and generate a large synthetic training data set using physically-based rendering. During testing, our network takes only the raw glossy images as input, without extra information such as segmentation masks or lighting estimation. Results demonstrate that multi-view reconstruction can be significantly improved using the images filtered by our network. We also show promising performance on real world training and testing data.