PhotoShape: Photorealistic Materials for Large-Scale Shape Collections

TL;DR

PhotoShape introduces an automated method to assign photorealistic, relightable materials to large-scale 3D shape collections by leveraging online data and neural networks, enhancing visual realism in 3D models.

Contribution

It presents a novel approach combining online data and neural networks to automatically generate photorealistic 3D shapes with realistic materials.

Findings

Successfully assigns realistic materials to large shape collections

Generates photorealistic, relightable 3D models

Uses neural networks trained on synthetic renderings

Abstract

Existing online 3D shape repositories contain thousands of 3D models but lack photorealistic appearance. We present an approach to automatically assign high-quality, realistic appearance models to large scale 3D shape collections. The key idea is to jointly leverage three types of online data -- shape collections, material collections, and photo collections, using the photos as reference to guide assignment of materials to shapes. By generating a large number of synthetic renderings, we train a convolutional neural network to classify materials in real photos, and employ 3D-2D alignment techniques to transfer materials to different parts of each shape model. Our system produces photorealistic, relightable, 3D shapes (PhotoShapes).