Efficient Reflectance Capture with a Deep Gated Mixture-of-Experts

TL;DR

This paper introduces a deep gated mixture-of-experts framework for efficient, high-quality reflectance capture that adapts to input conditions and reduces bandwidth requirements, validated on a near-field lightstage.

Contribution

It proposes a novel input-conditioned mixture-of-experts network with specialized decoders and a latent transform, improving reflectance reconstruction efficiency and quality over existing methods.

Findings

Achieves higher quality reflectance reconstruction at the same bandwidth.

Reduces bandwidth to about one-third while maintaining quality.

Validated on a challenging near-field lightstage dataset.

Abstract

We present a novel framework to efficiently acquire near-planar anisotropic reflectance in a pixel-independent fashion, using a deep gated mixtureof-experts. While existing work employs a unified network to handle all possible input, our network automatically learns to condition on the input for enhanced reconstruction. We train a gating module to select one out of a number of specialized decoders for reflectance reconstruction, based on photometric measurements, essentially trading generality for quality. A common, pre-trained latent transform module is also appended to each decoder, to offset the burden of the increased number of decoders. In addition, the illumination conditions during acquisition can be jointly optimized. The effectiveness of our framework is validated on a wide variety of challenging samples using a near-field lightstage. Compared with the state-of-the-art technique, our results are improved at the same input bandwidth, and our bandwidth can be reduced to about 1/3 for equal-quality results.