Shadow Transfer: Single Image Relighting For Urban Road Scenes

TL;DR

Shadow Transfer is a self-supervised deep learning framework that relights outdoor scenes in a single image by transferring realistic shadows and lighting effects, improving visual quality and potentially aiding urban scene understanding.

Contribution

It introduces a novel self-supervised approach for relighting outdoor scenes that leverages sensor and label data, addressing limitations of existing image translation methods.

Findings

Produces higher quality relighted images than state-of-the-art methods.

Effectively transfers realistic shadows and lighting effects onto single images.

Demonstrates applicability on both synthetic and real datasets.

Abstract

Illumination effects in images, specifically cast shadows and shading, have been shown to decrease the performance of deep neural networks on a large number of vision-based detection, recognition and segmentation tasks in urban driving scenes. A key factor that contributes to this performance gap is the lack of `time-of-day' diversity within real, labeled datasets. There have been impressive advances in the realm of image to image translation in transferring previously unseen visual effects into a dataset, specifically in day to night translation. However, it is not easy to constrain what visual effects, let alone illumination effects, are transferred from one dataset to another during the training process. To address this problem, we propose deep learning framework, called Shadow Transfer, that can relight complex outdoor scenes by transferring realistic shadow, shading, and other lighting effects onto a single image. The novelty of the proposed framework is that it is both self-supervised, and is designed to operate on sensor and label information that is easily available in autonomous vehicle datasets. We show the effectiveness of this method on both synthetic and real datasets, and we provide experiments that demonstrate that the proposed method produces images of higher visual quality than state of the art image to image translation methods.