Understanding Untrained Deep Models for Inverse Problems: Algorithms and Theory

TL;DR

This paper reviews the Deep Image Prior (DIP) method for inverse imaging problems, analyzing its training dynamics, recent improvements to reduce overfitting, and comparing it with data-driven approaches, highlighting future research directions.

Contribution

It provides a comprehensive theoretical and empirical review of DIP, including recent advancements to mitigate overfitting and hybrid methods with pre-trained networks.

Findings

DIP can effectively restore images without training data.

Overfitting remains a key challenge in DIP.

Recent techniques improve DIP's robustness and performance.

Abstract

In recent years, deep learning methods have been extensively developed for inverse imaging problems (IIPs), encompassing supervised, self-supervised, and generative approaches. Most of these methods require large amounts of labeled or unlabeled training data to learn effective models. However, in many practical applications, such as medical image reconstruction, extensive training datasets are often unavailable or limited. A significant milestone in addressing this challenge came in 2018 with the work of Ulyanov et al., which introduced the Deep Image Prior (DIP)--the first training-data-free neural network method for IIPs. Unlike conventional deep learning approaches, DIP requires only a convolutional neural network, the noisy measurements, and a forward operator. By leveraging the implicit regularization of deep networks initialized with random noise, DIP can learn and restore image structures without relying on external datasets. However, a well-known limitation of DIP is its susceptibility to overfitting, primarily due to the over-parameterization of the network. In this tutorial paper, we provide a comprehensive review of DIP, including a theoretical analysis of its training dynamics. We also categorize and discuss recent advancements in DIP-based methods aimed at mitigating overfitting, including techniques such as regularization, network re-parameterization, and early stopping. Furthermore, we discuss approaches that combine DIP with pre-trained neural networks, present empirical comparison results against data-centric methods, and highlight open research questions and future directions.