360U-Former: HDR Illumination Estimation with Panoramic Adapted Vision Transformers

TL;DR

This paper introduces 360U-Former, a novel Vision-Transformer-based architecture tailored for HDR illumination estimation from panoramas, effectively reducing artifacts and improving accuracy over existing methods.

Contribution

It presents the first pure Vision-Transformer model for illumination estimation, adapting shifted window attention to the ERP format and training it as a GAN for HDRI generation.

Findings

Outperforms existing methods on standard datasets

Reduces seam artifacts in HDRI results

Demonstrates effectiveness of Vision-Transformer in illumination estimation

Abstract

Recent illumination estimation methods have focused on enhancing the resolution and improving the quality and diversity of the generated textures. However, few have explored tailoring the neural network architecture to the Equirectangular Panorama (ERP) format utilised in image-based lighting. Consequently, high dynamic range images (HDRI) results usually exhibit a seam at the side borders and textures or objects that are warped at the poles. To address this shortcoming we propose a novel architecture, 360U-Former, based on a U-Net style Vision-Transformer which leverages the work of PanoSWIN, an adapted shifted window attention tailored to the ERP format. To the best of our knowledge, this is the first purely Vision-Transformer model used in the field of illumination estimation. We train 360U-Former as a GAN to generate HDRI from a limited field of view low dynamic range image (LDRI). We evaluate our method using current illumination estimation evaluation protocols and datasets, demonstrating that our approach outperforms existing and state-of-the-art methods without the artefacts typically associated with the use of the ERP format.