Spherical Image Inpainting with Frame Transformation and Data-driven Prior Deep Networks

TL;DR

This paper introduces a novel spherical image inpainting method combining framelet transforms, a deep CNN denoiser, and plug-and-play optimization, significantly improving recovery of damaged spherical images.

Contribution

It develops a new framework using directional spherical Haar framelets and a progressive CNN denoiser for effective spherical image inpainting.

Findings

Achieves superior inpainting performance over existing methods.

Effectively recovers heavily damaged spherical images.

Demonstrates efficiency of plug-and-play with CNN prior.

Abstract

Spherical image processing has been widely applied in many important fields, such as omnidirectional vision for autonomous cars, global climate modelling, and medical imaging. It is non-trivial to extend an algorithm developed for flat images to the spherical ones. In this work, we focus on the challenging task of spherical image inpainting with deep learning-based regularizer. Instead of a naive application of existing models for planar images, we employ a fast directional spherical Haar framelet transform and develop a novel optimization framework based on a sparsity assumption of the framelet transform. Furthermore, by employing progressive encoder-decoder architecture, a new and better-performed deep CNN denoiser is carefully designed and works as an implicit regularizer. Finally, we use a plug-and-play method to handle the proposed optimization model, which can be implemented efficiently by training the CNN denoiser prior. Numerical experiments are conducted and show that the proposed algorithms can greatly recover damaged spherical images and achieve the best performance over purely using deep learning denoiser and plug-and-play model.