OutCast: Outdoor Single-image Relighting with Cast Shadows

TL;DR

OutCast introduces a novel outdoor image relighting technique that predicts cast shadows from a single image by converting approximate depth maps into deep 3D representations, enabling state-of-the-art results without multiple images.

Contribution

This work presents a new method that uses a learned image space ray-marching layer to improve shadow prediction from single-image depth estimates for outdoor relighting.

Findings

Achieves state-of-the-art outdoor relighting results from a single image.

Effectively predicts cast shadows and global lighting effects.

Uses a learned 3D representation to enhance depth-based shadow rendering.

Abstract

We propose a relighting method for outdoor images. Our method mainly focuses on predicting cast shadows in arbitrary novel lighting directions from a single image while also accounting for shading and global effects such the sun light color and clouds. Previous solutions for this problem rely on reconstructing occluder geometry, e.g. using multi-view stereo, which requires many images of the scene. Instead, in this work we make use of a noisy off-the-shelf single-image depth map estimation as a source of geometry. Whilst this can be a good guide for some lighting effects, the resulting depth map quality is insufficient for directly ray-tracing the shadows. Addressing this, we propose a learned image space ray-marching layer that converts the approximate depth map into a deep 3D representation that is fused into occlusion queries using a learned traversal. Our proposed method achieves, for the first time, state-of-the-art relighting results, with only a single image as input. For supplementary material visit our project page at: https://dgriffiths.uk/outcast.