Optimal Eye Surgeon: Finding Image Priors through Sparse Generators at Initialization

TL;DR

Optimal Eye Surgeon (OES) is a novel framework that adaptively prunes untrained deep image generator networks at initialization, creating effective image priors that resist overfitting and improve image restoration tasks.

Contribution

OES introduces a new method for pruning networks at initialization to produce underparameterized subnetworks that serve as robust image priors, outperforming existing pruning techniques.

Findings

Pruned subnetworks effectively capture low-frequency image components.

OES masks transfer well across different models and tasks.

Sparse subnetworks resist overfitting to noise in image restoration.

Abstract

We introduce Optimal Eye Surgeon (OES), a framework for pruning and training deep image generator networks. Typically, untrained deep convolutional networks, which include image sampling operations, serve as effective image priors (Ulyanov et al., 2018). However, they tend to overfit to noise in image restoration tasks due to being overparameterized. OES addresses this by adaptively pruning networks at random initialization to a level of underparameterization. This process effectively captures low-frequency image components even without training, by just masking. When trained to fit noisy images, these pruned subnetworks, which we term Sparse-DIP, resist overfitting to noise. This benefit arises from underparameterization and the regularization effect of masking, constraining them in the manifold of image priors. We demonstrate that subnetworks pruned through OES surpass other leading pruning methods, such as the Lottery Ticket Hypothesis, which is known to be suboptimal for image recovery tasks (Wu et al., 2023). Our extensive experiments demonstrate the transferability of OES-masks and the characteristics of sparse-subnetworks for image generation. Code is available at https://github.com/Avra98/Optimal-Eye-Surgeon.git.