PanoViT: Vision Transformer for Room Layout Estimation from a Single Panoramic Image

TL;DR

PanoViT is a novel vision transformer model designed for accurate room layout estimation from single panoramic images, outperforming existing methods by effectively capturing global and local features.

Contribution

The paper introduces PanoViT with a unique recurrent position embedding and patch sampling tailored for panoramic images, enhancing global and local feature extraction.

Findings

Outperforms state-of-the-art in room layout prediction

Effective global and local feature extraction from panoramas

Improved accuracy on multiple datasets

Abstract

In this paper, we propose PanoViT, a panorama vision transformer to estimate the room layout from a single panoramic image. Compared to CNN models, our PanoViT is more proficient in learning global information from the panoramic image for the estimation of complex room layouts. Considering the difference between a perspective image and an equirectangular image, we design a novel recurrent position embedding and a patch sampling method for the processing of panoramic images. In addition to extracting global information, PanoViT also includes a frequency-domain edge enhancement module and a 3D loss to extract local geometric features in a panoramic image. Experimental results on several datasets demonstrate that our method outperforms state-of-the-art solutions in room layout prediction accuracy.