Semantically Supervised Appearance Decomposition for Virtual Staging from a Single Panorama

TL;DR

This paper introduces a weakly supervised, GAN-based method for decomposing a single indoor panorama into four appearance components, enabling virtual staging applications like furniture insertion and lighting adjustments.

Contribution

It presents a novel approach that uses semantic maps and transfer learning to decompose panoramas without extensive annotations, advancing virtual home staging technology.

Findings

Effective decomposition of panoramas into four appearance components.

Successful removal of lighting effects for virtual modifications.

Demonstrated applications include furniture insertion and sun direction change.

Abstract

We describe a novel approach to decompose a single panorama of an empty indoor environment into four appearance components: specular, direct sunlight, diffuse and diffuse ambient without direct sunlight. Our system is weakly supervised by automatically generated semantic maps (with floor, wall, ceiling, lamp, window and door labels) that have shown success on perspective views and are trained for panoramas using transfer learning without any further annotations. A GAN-based approach supervised by coarse information obtained from the semantic map extracts specular reflection and direct sunlight regions on the floor and walls. These lighting effects are removed via a similar GAN-based approach and a semantic-aware inpainting step. The appearance decomposition enables multiple applications including sun direction estimation, virtual furniture insertion, floor material replacement, and sun direction change, providing an effective tool for virtual home staging. We demonstrate the effectiveness of our approach on a large and recently released dataset of panoramas of empty homes.