Empty Cities: Image Inpainting for a Dynamic-Object-Invariant Space

TL;DR

This paper introduces a deep learning framework that removes dynamic objects from images to produce realistic static scenes, aiding applications like augmented reality and robot localization.

Contribution

It presents a novel combination of semantic segmentation and generative adversarial networks for dynamic object removal and background inpainting in images.

Findings

Outperforms state-of-the-art inpainting methods in qualitative and quantitative tests.

Produces realistic static images from dynamic scenes.

Enhances visual place recognition capabilities.

Abstract

In this paper we present an end-to-end deep learning framework to turn images that show dynamic content, such as vehicles or pedestrians, into realistic static frames. This objective encounters two main challenges: detecting all the dynamic objects, and inpainting the static occluded background with plausible imagery. The second problem is approached with a conditional generative adversarial model that, taking as input the original dynamic image and its dynamic/static binary mask, is capable of generating the final static image. The former challenge is addressed by the use of a convolutional network that learns a multi-class semantic segmentation of the image. These generated images can be used for applications such as augmented reality or vision-based robot localization purposes. To validate our approach, we show both qualitative and quantitative comparisons against other state-of-the-art inpainting methods by removing the dynamic objects and hallucinating the static structure behind them. Furthermore, to demonstrate the potential of our results, we carry out pilot experiments that show the benefits of our proposal for visual place recognition.