Degrees of Freedom Analysis of Unrolled Neural Networks

TL;DR

This paper analyzes the generalization risk of unrolled neural networks in image restoration, linking degrees-of-freedom to network design and training data, and finds recurrence acts as a regularizer for small sample sizes.

Contribution

It introduces a SURE-based analysis of degrees-of-freedom in unrolled networks, revealing how recurrence influences generalization and sample efficiency.

Findings

DOF increases with training sample size and approaches generalization risk.

Recurrent networks converge faster and require fewer samples than non-recurrent ones.

Recurrence acts as a regularizer in low-sample regimes.

Abstract

Unrolled neural networks emerged recently as an effective model for learning inverse maps appearing in image restoration tasks. However, their generalization risk (i.e., test mean-squared-error) and its link to network design and train sample size remains mysterious. Leveraging the Stein's Unbiased Risk Estimator (SURE), this paper analyzes the generalization risk with its bias and variance components for recurrent unrolled networks. We particularly investigate the degrees-of-freedom (DOF) component of SURE, trace of the end-to-end network Jacobian, to quantify the prediction variance. We prove that DOF is well-approximated by the weighted \textit{path sparsity} of the network under incoherence conditions on the trained weights. Empirically, we examine the SURE components as a function of train sample size for both recurrent and non-recurrent (with many more parameters) unrolled networks. Our key observations indicate that: 1) DOF increases with train sample size and converges to the generalization risk for both recurrent and non-recurrent schemes; 2) recurrent network converges significantly faster (with less train samples) compared with non-recurrent scheme, hence recurrence serves as a regularization for low sample size regimes.