Early Stopping for Deep Image Prior

TL;DR

This paper introduces an effective early stopping strategy for Deep Image Prior models that detects near-peak performance across various vision tasks, improving practicality without requiring ground truth access.

Contribution

The paper proposes a simple dispersion-based early stopping method for DIP that outperforms existing techniques and is broadly applicable across different vision tasks and DIP variants.

Findings

The proposed ES method consistently detects near-peak performance.

It outperforms existing early stopping techniques in various tasks.

The method remains effective when combined with overfitting mitigation strategies.

Abstract

Deep image prior (DIP) and its variants have showed remarkable potential for solving inverse problems in computer vision, without any extra training data. Practical DIP models are often substantially overparameterized. During the fitting process, these models learn mostly the desired visual content first, and then pick up the potential modeling and observational noise, i.e., overfitting. Thus, the practicality of DIP often depends critically on good early stopping (ES) that captures the transition period. In this regard, the majority of DIP works for vision tasks only demonstrates the potential of the models -- reporting the peak performance against the ground truth, but provides no clue about how to operationally obtain near-peak performance without access to the groundtruth. In this paper, we set to break this practicality barrier of DIP, and propose an efficient ES strategy, which consistently detects near-peak performance across several vision tasks and DIP variants. Based on a simple measure of dispersion of consecutive DIP reconstructions, our ES method not only outpaces the existing ones -- which only work in very narrow domains, but also remains effective when combined with a number of methods that try to mitigate the overfitting. The code is available at https://github.com/sun-umn/Early_Stopping_for_DIP.