SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage

TL;DR

SunStage offers a low-cost, smartphone-based method for capturing facial data outdoors using sunlight, enabling detailed 3D face reconstruction and relighting without specialized equipment.

Contribution

It introduces a novel outdoor selfie video approach leveraging the sun's movement for facial reconstruction, eliminating the need for expensive light stage setups.

Findings

Achieves detailed facial geometry and reflectance reconstruction.

Enables relighting and novel view synthesis in uncontrolled outdoor settings.

Operates effectively with uncalibrated, in-the-wild data.

Abstract

A light stage uses a series of calibrated cameras and lights to capture a subject's facial appearance under varying illumination and viewpoint. This captured information is crucial for facial reconstruction and relighting. Unfortunately, light stages are often inaccessible: they are expensive and require significant technical expertise for construction and operation. In this paper, we present SunStage: a lightweight alternative to a light stage that captures comparable data using only a smartphone camera and the sun. Our method only requires the user to capture a selfie video outdoors, rotating in place, and uses the varying angles between the sun and the face as guidance in joint reconstruction of facial geometry, reflectance, camera pose, and lighting parameters. Despite the in-the-wild un-calibrated setting, our approach is able to reconstruct detailed facial appearance and geometry, enabling compelling effects such as relighting, novel view synthesis, and reflectance editing. Results and interactive demos are available at https://sunstage.cs.washington.edu/.