BIPS: Bi-modal Indoor Panorama Synthesis via Residual Depth-aided Adversarial Learning

TL;DR

This paper introduces BIPS, a novel framework for synthesizing high-quality indoor RGB-D panoramas from limited scene information using residual depth-aided adversarial learning, enabling realistic 3D indoor modeling.

Contribution

It proposes a new bi-modal panorama synthesis method for indoor environments, integrating RGB and depth data with residual-aided adversarial training for improved realism.

Findings

Synthesizes realistic indoor RGB-D panoramas

Provides a new metric for evaluating RGB-D panorama quality

Outperforms prior methods in generating 3D indoor models

Abstract

Providing omnidirectional depth along with RGB information is important for numerous applications, eg, VR/AR. However, as omnidirectional RGB-D data is not always available, synthesizing RGB-D panorama data from limited information of a scene can be useful. Therefore, some prior works tried to synthesize RGB panorama images from perspective RGB images; however, they suffer from limited image quality and can not be directly extended for RGB-D panorama synthesis. In this paper, we study a new problem: RGB-D panorama synthesis under the arbitrary configurations of cameras and depth sensors. Accordingly, we propose a novel bi-modal (RGB-D) panorama synthesis (BIPS) framework. Especially, we focus on indoor environments where the RGB-D panorama can provide a complete 3D model for many applications. We design a generator that fuses the bi-modal information and train it with residual-aided adversarial learning (RDAL). RDAL allows to synthesize realistic indoor layout structures and interiors by jointly inferring RGB panorama, layout depth, and residual depth. In addition, as there is no tailored evaluation metric for RGB-D panorama synthesis, we propose a novel metric to effectively evaluate its perceptual quality. Extensive experiments show that our method synthesizes high-quality indoor RGB-D panoramas and provides realistic 3D indoor models than prior methods. Code will be released upon acceptance.