Repopulating Street Scenes

TL;DR

This paper introduces a data-driven framework for automatically editing street scene images by adding or removing objects like pedestrians and vehicles, enabling applications such as privacy, data augmentation, and scene composition.

Contribution

It presents novel methods for object removal, sun direction estimation, and scene-aware object composition, all learned with minimal annotations from street scene image bursts.

Findings

Effective object depopulation and repopulation in street scenes

Accurate sun direction estimation from single images

Versatile scene editing demonstrated on diverse street scenes

Abstract

We present a framework for automatically reconfiguring images of street scenes by populating, depopulating, or repopulating them with objects such as pedestrians or vehicles. Applications of this method include anonymizing images to enhance privacy, generating data augmentations for perception tasks like autonomous driving, and composing scenes to achieve a certain ambiance, such as empty streets in the early morning. At a technical level, our work has three primary contributions: (1) a method for clearing images of objects, (2) a method for estimating sun direction from a single image, and (3) a way to compose objects in scenes that respects scene geometry and illumination. Each component is learned from data with minimal ground truth annotations, by making creative use of large-numbers of short image bursts of street scenes. We demonstrate convincing results on a range of street scenes and illustrate potential applications.