SemIE: Semantically-aware Image Extrapolation

TL;DR

This paper introduces SemIE, a semantically-aware image extrapolation method that extends existing images by adding new objects based on contextual understanding, outperforming previous approaches in quality and realism.

Contribution

The paper presents a novel paradigm for image extrapolation that not only extends existing objects but also adds new objects using semantic segmentation and panoptic segmentation.

Findings

Outperforms baselines in FID scores

Better preserves object co-occurrence statistics

Effective on Cityscapes and ADE20K-bedroom datasets

Abstract

We propose a semantically-aware novel paradigm to perform image extrapolation that enables the addition of new object instances. All previous methods are limited in their capability of extrapolation to merely extending the already existing objects in the image. However, our proposed approach focuses not only on (i) extending the already present objects but also on (ii) adding new objects in the extended region based on the context. To this end, for a given image, we first obtain an object segmentation map using a state-of-the-art semantic segmentation method. The, thus, obtained segmentation map is fed into a network to compute the extrapolated semantic segmentation and the corresponding panoptic segmentation maps. The input image and the obtained segmentation maps are further utilized to generate the final extrapolated image. We conduct experiments on Cityscapes and ADE20K-bedroom datasets and show that our method outperforms all baselines in terms of FID, and similarity in object co-occurrence statistics.