Deep Inverse Shading: Consistent Albedo and Surface Detail Recovery via Generative Refinement

TL;DR

DIS introduces a unified surface-based framework that combines generative priors, differentiable rasterization, and de-shading to achieve high-fidelity, relightable human avatar reconstructions with detailed geometry and material properties.

Contribution

The paper presents a novel mesh-based method integrating generative normal predictions and de-shading for detailed, relightable avatar reconstruction, overcoming vertex count limitations and improving efficiency.

Findings

Achieves state-of-the-art relighting quality.

Enhances rendering efficiency and reduces memory usage.

Provides detailed surface reconstruction beyond sparse vertex limitations.

Abstract

Reconstructing human avatars using generative priors is essential for achieving versatile and realistic avatar models. Traditional approaches often rely on volumetric representations guided by generative models, but these methods require extensive volumetric rendering queries, leading to slow training. Alternatively, surface-based representations offer faster optimization through differentiable rasterization, yet they are typically limited by vertex count, restricting mesh resolution and scalability when combined with generative priors. Moreover, integrating generative priors into physically based human avatar modeling remains largely unexplored. To address these challenges, we introduce DIS (Deep Inverse Shading), a unified framework for high-fidelity, relightable avatar reconstruction that incorporates generative priors into a coherent surface representation. DIS centers on a mesh-based model that serves as the target for optimizing both surface and material details. The framework fuses multi-view 2D generative surface normal predictions, rich in detail but often inconsistent, into the central mesh using a normal conversion module. This module converts generative normal outputs into per-triangle surface offsets via differentiable rasterization, enabling the capture of fine geometric details beyond sparse vertex limitations. Additionally, DIS integrates a de-shading module to recover accurate material properties. This module refines albedo predictions by removing baked-in shading and back-propagates reconstruction errors to optimize the geometry. Through joint optimization of geometry and material appearance, DIS achieves physically consistent, high-quality reconstructions suitable for accurate relighting. Our experiments show that DIS delivers SOTA relighting quality, enhanced rendering efficiency, lower memory consumption, and detailed surface reconstruction.