OSRT: Omnidirectional Image Super-Resolution with Distortion-aware Transformer

TL;DR

This paper introduces OSRT, a distortion-aware Transformer for omnidirectional image super-resolution that models realistic degradation and uses data augmentation to achieve state-of-the-art results.

Contribution

The paper proposes a realistic fisheye downsampling method and a distortion-aware Transformer for improved omnidirectional image super-resolution.

Findings

OSRT outperforms previous methods by about 0.2dB on PSNR.

Fisheye downsampling better mimics real-world imaging.

Data augmentation significantly boosts performance.

Abstract

Omnidirectional images (ODIs) have obtained lots of research interest for immersive experiences. Although ODIs require extremely high resolution to capture details of the entire scene, the resolutions of most ODIs are insufficient. Previous methods attempt to solve this issue by image super-resolution (SR) on equirectangular projection (ERP) images. However, they omit geometric properties of ERP in the degradation process, and their models can hardly generalize to real ERP images. In this paper, we propose Fisheye downsampling, which mimics the real-world imaging process and synthesizes more realistic low-resolution samples. Then we design a distortion-aware Transformer (OSRT) to modulate ERP distortions continuously and self-adaptively. Without a cumbersome process, OSRT outperforms previous methods by about 0.2dB on PSNR. Moreover, we propose a convenient data augmentation strategy, which synthesizes pseudo ERP images from plain images. This simple strategy can alleviate the over-fitting problem of large networks and significantly boost the performance of ODISR. Extensive experiments have demonstrated the state-of-the-art performance of our OSRT. Codes and models will be available at https://github.com/Fanghua-Yu/OSRT.