PanoSwin: a Pano-style Swin Transformer for Panorama Understanding

TL;DR

PanoSwin introduces a novel Swin Transformer architecture tailored for panorama understanding, addressing equirectangular projection challenges with specialized windowing, attention mechanisms, and a two-stage learning framework, achieving state-of-the-art results.

Contribution

The paper presents PanoSwin, a new architecture with pano-style shift windowing, pitch attention, and knowledge transfer from planar images for improved panorama understanding.

Findings

Outperforms existing methods on panoramic object detection.

Effective in panoramic classification tasks.

Enhances layout estimation accuracy.

Abstract

In panorama understanding, the widely used equirectangular projection (ERP) entails boundary discontinuity and spatial distortion. It severely deteriorates the conventional CNNs and vision Transformers on panoramas. In this paper, we propose a simple yet effective architecture named PanoSwin to learn panorama representations with ERP. To deal with the challenges brought by equirectangular projection, we explore a pano-style shift windowing scheme and novel pitch attention to address the boundary discontinuity and the spatial distortion, respectively. Besides, based on spherical distance and Cartesian coordinates, we adapt absolute positional embeddings and relative positional biases for panoramas to enhance panoramic geometry information. Realizing that planar image understanding might share some common knowledge with panorama understanding, we devise a novel two-stage learning framework to facilitate knowledge transfer from the planar images to panoramas. We conduct experiments against the state-of-the-art on various panoramic tasks, i.e., panoramic object detection, panoramic classification, and panoramic layout estimation. The experimental results demonstrate the effectiveness of PanoSwin in panorama understanding.