Minimalist and High-Quality Panoramic Imaging with PSF-aware Transformers

TL;DR

This paper introduces a novel minimalist panoramic imaging system using PSF-aware transformers to produce high-quality 360-degree images with fewer lenses, leveraging PSF maps for aberration correction and super-resolution.

Contribution

It proposes a new Panoramic Computational Imaging Engine with a PSF-aware transformer that enhances image quality in minimalist systems, filling a gap in real-world high-quality panoramic imaging.

Findings

Effective aberration correction and super-resolution achieved.

High-quality panoramic images demonstrated on synthetic and real datasets.

The PSF-aware transformer outperforms traditional methods.

Abstract

High-quality panoramic images with a Field of View (FoV) of 360{\deg} are essential for contemporary panoramic computer vision tasks. However, conventional imaging systems come with sophisticated lens designs and heavy optical components. This disqualifies their usage in many mobile and wearable applications where thin and portable, minimalist imaging systems are desired. In this paper, we propose a Panoramic Computational Imaging Engine (PCIE) to achieve minimalist and high-quality panoramic imaging. With less than three spherical lenses, a Minimalist Panoramic Imaging Prototype (MPIP) is constructed based on the design of the Panoramic Annular Lens (PAL), but with low-quality imaging results due to aberrations and small image plane size. We propose two pipelines, i.e. Aberration Correction (AC) and Super-Resolution and Aberration Correction (SR&AC), to solve the image quality problems of MPIP, with imaging sensors of small and large pixel size, respectively. To leverage the prior information of the optical system, we propose a Point Spread Function (PSF) representation method to produce a PSF map as an additional modality. A PSF-aware Aberration-image Recovery Transformer (PART) is designed as a universal network for the two pipelines, in which the self-attention calculation and feature extraction are guided by the PSF map. We train PART on synthetic image pairs from simulation and put forward the PALHQ dataset to fill the gap of real-world high-quality PAL images for low-level vision. A comprehensive variety of experiments on synthetic and real-world benchmarks demonstrates the impressive imaging results of PCIE and the effectiveness of the PSF representation. We further deliver heuristic experimental findings for minimalist and high-quality panoramic imaging. Our dataset and code will be available at https://github.com/zju-jiangqi/PCIE-PART.