On Measuring and Controlling the Spectral Bias of the Deep Image Prior

TL;DR

This paper analyzes the spectral bias of the deep image prior, introduces measures to control it, and proposes architectural modifications to improve training stability and performance across various image restoration tasks.

Contribution

It introduces a frequency-band correspondence measure for spectral bias and proposes new layers to control it, improving training stability without oracle stopping.

Findings

Spectral bias causes low-frequency signals to learn faster.

Proposed layers prevent performance degradation during training.

Method achieves better results in denoising, inpainting, and super-resolution.

Abstract

The deep image prior showed that a randomly initialized network with a suitable architecture can be trained to solve inverse imaging problems by simply optimizing it's parameters to reconstruct a single degraded image. However, it suffers from two practical limitations. First, it remains unclear how to control the prior beyond the choice of the network architecture. Second, training requires an oracle stopping criterion as during the optimization the performance degrades after reaching an optimum value. To address these challenges we introduce a frequency-band correspondence measure to characterize the spectral bias of the deep image prior, where low-frequency image signals are learned faster and better than high-frequency counterparts. Based on our observations, we propose techniques to prevent the eventual performance degradation and accelerate convergence. We introduce a Lipschitz-controlled convolution layer and a Gaussian-controlled upsampling layer as plug-in replacements for layers used in the deep architectures. The experiments show that with these changes the performance does not degrade during optimization, relieving us from the need for an oracle stopping criterion. We further outline a stopping criterion to avoid superfluous computation. Finally, we show that our approach obtains favorable results compared to current approaches across various denoising, deblocking, inpainting, super-resolution and detail enhancement tasks. Code is available at \url{https://github.com/shizenglin/Measure-and-Control-Spectral-Bias}.