Momentum-Net: Fast and convergent iterative neural network for inverse problems
Il Yong Chun, Zhengyu Huang, Hongki Lim, Jeffrey A. Fessler
TL;DR
Momentum-Net is a novel iterative neural network architecture that guarantees convergence and significantly enhances the speed and accuracy of inverse problem solutions in imaging applications.
Contribution
It introduces a fast, convergent INN architecture that generalizes block-wise MBIR algorithms using momentum and majorizers, with proven convergence guarantees.
Findings
Outperforms existing INNs in speed and accuracy.
Improves reconstruction quality over state-of-the-art MBIR methods.
Demonstrates effectiveness in light-field photography and tomography.
Abstract
Iterative neural networks (INN) are rapidly gaining attention for solving inverse problems in imaging, image processing, and computer vision. INNs combine regression NNs and an iterative model-based image reconstruction (MBIR) algorithm, often leading to both good generalization capability and outperforming reconstruction quality over existing MBIR optimization models. This paper proposes the first fast and convergent INN architecture, Momentum-Net, by generalizing a block-wise MBIR algorithm that uses momentum and majorizers with regression NNs. For fast MBIR, Momentum-Net uses momentum terms in extrapolation modules, and noniterative MBIR modules at each iteration by using majorizers, where each iteration of Momentum-Net consists of three core modules: image refining, extrapolation, and MBIR. Momentum-Net guarantees convergence to a fixed-point for general differentiable (non)convex…
| (a) FBP | (b) EP reg. | (c) Learned convolutional reg. [Chun&Fessler:20TIP:supp, Chun&Fessler:18Asilomar:supp] |
|
|
|
|||||||
| Test | 82.8 | 40.8 | 35.2 | 19.9 | 19.5 | 19.8 | ||||||
| Test | 74.9 | 38.5 | 34.5 | 18.4 | 17.7 | 17.8 |
| (a) | (b) 4D EP reg. [Lien&etal:20NP:supp] |
|
|
|||
|---|---|---|---|---|---|---|
| Test | 16.4 | 32.0 | 35.8 | 37.1 | ||
| Test | 21.1 | 28.1 | 30.7 | 32.0 | ||
| Test | 21.6 | 28.1 | 30.9 | 31.7 |
|
|
|
|
|
|||||||||||
| Test | 4.7 | 41.0 | 13.8 | 8.0 | 5.7 | ||||||||||
| Test | 30.5 | 117.6 | 39.5 | 34.6 | 31.9 | ||||||||||
| Test | n/a† | n/a† | n/a† | n/a† | n/a† |
| Param. | Module | Guidelines or default values |
|---|---|---|
| (Alg.1.1) | Trainable by §3.1. For selecting their architecture/param., see guideline §A.10. | |
| (Alg.1.1) | Use regularization strength ; see guideline in §A.10. | |
| in (8)–(9) | (Alg.1.2) | |
| (5) | Designed off-line. For large-scale inverse problems with quadratic data-fit, use Lemma A.7. | |
| in (7) | (5) | For convex , ; for nonconvex , . |
| (5) | Chosen by tuning/training in §3.2 |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsSPEED: Separable Pyramidal Pooling EncodEr-Decoder for Real-Time Monocular Depth Estimation on Low-Resource Settings
\newcites
SuppReferences
Momentum-Net: Fast and convergent
iterative neural network for inverse problems
Il Yong Chun, Zhengyu Huang∗, Hongki Lim∗,
and Jeffrey A. Fessler,
The authors indicated by asterisks (∗) contributed equally to this work. This work is supported in part by the Keck Foundation, NIH U01 EB018753, NIH R01 EB023618, NIH R01 EB022075, and NSF IIS 1838179. Il Yong Chun was with the Department of Electrical Engineering and Computer Science, The University of Michigan, Ann Arbor, MI 48019 USA, and is now with the Department of Electrical Engineering, the University of Hawai’i at Mānoa, Honolulu, HI 96822 USA (email: [email protected]). Zhengyu Huang, Hongki Lim, and Jeffrey A. Fessler are with the Department of Electrical Engineering and Computer Science, The University of Michigan, Ann Arbor, MI 48019 USA (email: [email protected]; [email protected]; [email protected]). This paper has appendices. The prefix “A” indicate the numbers in section, theorem, equation, figure, table, and footnote in the appendices. This paper was presented in part at the Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA, in Oct. 2018.
Abstract
Iterative neural networks (INN) are rapidly gaining attention for solving inverse problems in imaging, image processing, and computer vision. INNs combine regression NNs and an iterative model-based image reconstruction (MBIR) algorithm, often leading to both good generalization capability and outperforming reconstruction quality over existing MBIR optimization models. This paper proposes the first fast and convergent INN architecture, Momentum-Net, by generalizing a block-wise MBIR algorithm that uses momentum and majorizers with regression NNs. For fast MBIR, Momentum-Net uses momentum terms in extrapolation modules, and noniterative MBIR modules at each iteration by using majorizers, where each iteration of Momentum-Net consists of three core modules: image refining, extrapolation, and MBIR. Momentum-Net guarantees convergence to a fixed-point for general differentiable (non)convex MBIR functions (or data-fit terms) and convex feasible sets, under two asymptomatic conditions. To consider data-fit variations across training and testing samples, we also propose a regularization parameter selection scheme based on the “spectral spread” of majorization matrices. Numerical experiments for light-field photography using a focal stack and sparse-view computational tomography demonstrate that, given identical regression NN architectures, Momentum-Net significantly improves MBIR speed and accuracy over several existing INNs; it significantly improves reconstruction quality compared to a state-of-the-art MBIR method in each application.
Index Terms:
Iterative neural network, deep learning, model-based image reconstruction, inverse problems, block proximal extrapolated gradient method, block coordinate descent method, light-field photography, X-ray computational tomography.
1 Introduction
Deep regression neural network (NN) methods have been actively studied for solving diverse inverse problems, due to their effectiveness at mapping noisy signals into clean signals. Examples include image denoising [1, 2, 3, 4], image deconvolution [5, 6], image super-resolution [7, 8], magnetic resonance imaging (MRI) [9, 10], X-ray computational tomography (CT) [11, 12, 13], and light-field (LF) photography [14, 15]. However, regression NNs with a greater mapping capability have increased overfitting/hallucination risks [16, 17, 18, 19]. An alternative approach to solving inverse problems is an iterative NN (INN) that combines regression NNs – called “refiners” or denoisers – with an unrolled iterative model-based image reconstruction (MBIR) algorithm [20, 21, 22, 23, 24, 25, 26, 27]. This alternative approach can regulate overfitting of regression NNs, by balancing physical data-fit of MBIR and prior information estimated by refining NNs [18, 16]. This “soft-refiner” approach has been successfully applied to several extreme imaging systems, e.g., highly undersampled MRI [20, 28, 29, 30, 25], low-dose or sparse-view CT [31, 24, 27, 16, 19], and low-count emission tomography [32, 33, 18, 34].
1.1 Notation
This section provides mathematical notations. We use to denote a function of given . We use to denote the -norm and write for the standard inner product on . The weighted -norm with a Hermitian positive definite matrix is denoted by . The Frobenius norm of a matrix is denoted by . , , and indicate the transpose, complex conjugate transpose (Hermitian transpose), and complex conjugate, respectively. denotes the conversion of a vector into a diagonal matrix or diagonal elements of a matrix into a vector. For (self-adjoint) matrices , the notation denotes that is a positive semi-definite matrix.
1.2 From block-wise optimization to INN
To recover signals from measurements , consider the following MBIR optimization problem:
[TABLE]
where is a set of feasible points, is data-fit function, is a regularization parameter, and is some high-quality approximation to the true unknown signal . The data-fit measures deviations of model-based predictions of from data , considering models of imaging physics (or image formation) and noise statistics in . In (P0), the signal recovery accuracy increases as the quality of improves [17, Prop. 3]; however, it is difficult to obtain such in practice. Alternatively, there has been a growing trend in learning sparsifying regularizers (e.g., convolutional regularizers [35, 24, 36, 37, 38]) from training datasets and applying the trained regularizers to the following block-wise MBIR problem: . Here, a learned regularizer quantifies consistency between and refined sparse signal via some learned operators . Recently, we have constructed INNs by generalizing the corresponding block-wise MBIR updates with regression NNs without convergence analysis [25, 27]. In existing INNs, two major challenges exist: convergence and acceleration.
1.3 Challenges in existing INNs: Convergence
Existing convergence analysis has some practical limitations. The original form of plug-and-play (PnP [39, 40, 41, 23]) is motivated by the alternating direction method of multipliers (ADMM [42]), and its fixed-point convergence has been analyzed with consensus equilibrium perspectives [23]. However, similar to ADMM, its practical convergence depends on how one selects ADMM penalty parameters. For example, [22] reported unstable convergence behaviors of PnP-ADMM with fixed ADMM parameters. To moderate this problem, [41] proposed a scheme that adaptively controls the ADMM parameters based on relative residuals. Similar to the residual balancing technique [42, §3.4.1], the scheme in [41] requires tuning initial parameters. Regularization by Denoising (RED [22]) is an alternative that moderates some such limitations. In particular, RED aims to make a clear connection between optimization and a denoiser , by defining its prior term by (scaled) . Nonetheless, [43] showed that many practical denoisers do not satisfy the Jacobian symmetry in [22], and proposed a less restrictive method, score-matching by denoising.
The convergence analysis of the INN inspired by the relaxed projected gradient descent (RPGD) method in [31] has the least restrictive conditions on the regression NN among the existing INNs. This method replaces the projector of a projected gradient descent method with an image refining NN. However, the RPGD-inspired INN directly applies an image refining NN to gradient descent updates of data-fit; thus, this INN relies heavily on the mapping performance of a refining NN and can have overfitting risks, similar to non-MBIR regression NNs, e.g., FBPConvNet [12]. In addition, it exploits the data-fit term only for the first few iterations [31, Fig. 5(c)]. We refer the perspective used in RPGD-inspired INN and its related works [44, 26] as “hard-refiner”: different from soft-refiners, these methods do not use a refining NN as a regularizer. More recently, [26] presented convergence analysis for an INN inspired by a proximal gradient descent method. However, their analysis is based on noiseless measurements, which is typically impractical.
Broadly speaking, existing convergence analysis largely depends on the (firmly) nonexpansive property of image refining NNs [23, 22, 43], [31, PGD], [26]. However, except for a single-hidden layer convolutional NN (CNN), it is yet unclear which analytical conditions guarantee the non-expansiveness of general refining NNs [27]. To guarantee convergence of INNs even when using possibly expansive image refining NNs, we proposed a method that normalizes the output signal of image refining NNs by their Lipschitz constants [27]. However, if one uses expansive NNs that are identical across iterations, it is difficult to obtain “best” image recovery with that normalization scheme. The spectral normalization based training [45, 46] can ensure the non-expansiveness of refining NNs by single-step power iteration. However, similar to the normalization method in [27], refining NNs trained with the spectral normalization method [46] degraded the image reconstruction accuracy for an INN using iteration-wise refining NNs [19]. In addition, there does not yet exist theoretical convergence results when refining NNs change across iterations, yet iteration-wise refining NNs are widely studied [20, 21, 28, 25]. Finally, existing analysis considers only a narrow class of data-fit terms: most analyses consider a quadratic function with a linear imaging model [31, 26] or more generally, a convex cost function [43, 23, 46] that can be minimized with a practical closed-form solution. No theoretical convergence results exist for general (non)convex data-fit terms, iteration-wise NN denoisers, and a general set of feasible points.
1.4 Challenges in existing INNs: Acceleration
Compared to non-MBIR regression NNs that do not exploit the data-fit in (P0), INNs require more computation because they consider the imaging physics. Computation increases as the imaging system or image formation model becomes larger-scale, e.g., LF photography from a focal stack, 3D CT, parallel MRI using many receive coils, and image super-resolution. Thus, acceleration becomes crucial for INNs.
First, consider the existing methods motivated by ADMM or block coordinate descent (BCD) method: examples include PnP-ADMM [41, 23], RED-ADMM [22, 43], MoDL [30], BCD-Net [25, 18, 16], etc. These methods can require multiple inner iterations to balance data-fit and prior information estimated by trained refining NNs, increasing total MBIR time. For example, in solving such problems, each outer iteration involves , where is given as in (P0) and is the output from the th image refining NN. For LF imaging system using a focal stack data [47], solving the above problem requires multiple iterations, and the total computational cost scale with the numbers of photosensors and sub-aperture images. In addition, nonconvexity of the data-fit term can break convergence guarantees of these methods, because in general, the proximal mapping is no longer nonexpansive.
Second, consider the existing works motivated by gradient descent methods [21, 28, 31, 26]. These methods resolve the inner iteration issue; however, they lack a sophisticated step-size control or backtracking scheme that influences convergence guarantee and acceleration. Accelerated proximal gradient (APG) methods using momentum terms can significantly accelerate convergence rates for solving composite convex problems [48, 49], so we expect that INN methods in the second class have yet to be maximally accelerated. The work in [44] applied PnP to the APG method [49]; [50] applied PnP to the primal-dual splitting (PDS) algorithm [51]. However, similar to RPGD [31], these are hard-refiner methods using some state-of-the-art denoisiers (e.g., BM3D [52]) but not trained NNs. Those methods lack convergence analyses and guarantees may be limited to convex data-fit function.
1.5 Contributions and organization of the paper
This paper proposes Momentum-Net, the first INN architecture that aims for fast and convergent MBIR. The architecture of Momentum-Net is motivated by applying the Block Proximal Extrapolated Gradient method using a Majorizer (BPEG-M) [35, 24] to MBIR using trainable convolutional autoencoders [24, 25, 37]. Specifically, each iteration of Momentum-Net consists of three core modules: image refining, extrapolation, and MBIR. At each Momentum-Net iteration, an extrapolation module uses momentum from previous updates to amplify the changes in subsequent iterations and accelerate convergence, and an MBIR module is noniterative. In addition, Momentum-Net resolves the convergence issues mentioned in §1.3: for general differentiable (non)convex data-fit terms and convex feasible sets, it guarantees convergence to a point that satisfies fixed-point and critical point conditions, under some mild conditions and two asymptotic conditions, i.e., asymptotically nonexpansive paired refining NNs and asymptotically block-coordinate minimizer.
The remainder of this paper is organized as follows. §2 constructs the Momentum-Net architecture motivated by BPEG-M algorithm that solves MBIR problem using a learnable convolutional regularizer, describes its relation to existing works, analyzes its convergence, and summarizes the benefits of Momentum-Net over existing INNs. §3 provides details of training INNs, including image refining NN architectures, single-hidden layer or “shallow” CNN (sCNN) and multi-hidden layer or “deep” CNN (dCNN), and training loss function, and proposes a regularization parameter selection scheme to consider data-fit variations across training and testing samples. §4 considers two extreme imaging applications: sparse-view CT and LF photography using a focal stack. §4 reports numerical experiments of applications where the proposed Momentum-Net using extrapolation significantly improves MBIR speed and accuracy, over the existing INNs, BCD-Net [25, 30, 22], Momentum-Net using no extrapolation [21, 28], ADMM-Net [20, 41, 23], and PnP-PDS [50] using refining NNs. Furthermore, §4 reports numerical experiments where Momentum-Net significantly improves reconstruction quality compared to a state-of-the-art MBIR method in each application.
2 Momentum-Net: Where BPEG-M meets NNs for inverse problems
2.1 Motivation: BPEG-M algorithm for MBIR using learnable convolutional regularizer
This section motivates the proposed Momentum-Net architecture, based on our previous works [24, 37]. Consider the following approach for recovering signal from measurements (see the setup of block multi-(non)convex problems in §A.1.1):
[TABLE]
where is a closed set, is a (continuosly) differentiable (non)convex function in , is a learnable convolutional regularizer [24, 36], is a set of sparse features that correspond to \{h_{k}\mathop{\scalebox{1.5}{\raisebox{-0.86108pt}{\ast}}}x\}, is a set of trainable filters, and and denote the size and number of trained filters, respectively.
Problem (2.1) can be viewed as a two-block optimization problem in terms of the image and the features . We solve (2.1) using the recent BPEG-M optimization framework [35, 24] that has attractive convergence guarantee and rapidly solved several block optimization problems [53, 54, 55, 35, 24]. BPEG-M has the following key ideas for each block optimization problem (see details in §A.1):
- •
-Lipschitz continuity for the gradient of the block optimization problem, :
Definition 1** (-Lipschitz continuity [24]).**
A function is -Lipschitz continuous on if there exists a (symmetric) positive definite matrix such that
[TABLE]
Definition 1 is a more general concept than the classical Lipschitz continuity.
- •
A sharper majorization matrix that gives a tighter bound in Definition 1 leads to a tighter quadratic majorization bound in the following lemma:
Lemma 2** (Quadratic majorization via -Lipschitz continuous gradients [24]).**
Let . If is -Lipschitz continuous, then
[TABLE]
Having tighter majorization bounds, sharper majorization matrices tend to accelerate BPEG-M convergence.
- •
The majorized block problems are “proximable”, i.e., proximal mapping of majorized function is “easily” computable depending on the properties of block majorizer and regularizer, and , where the proximal mapping operator is defined by
[TABLE]
- •
Block-wise extrapolation and momentum terms to accelerate convergence.
Suppose that 1) gradient of is -Lipschitz continuous at an extrapolated point , ; 2) filters in (2.1) satisfy the tight-frame (TF) condition, \sum_{k=1}^{K}{\left\|h_{k}\mathop{\scalebox{1.5}{\raisebox{-0.86108pt}{\ast}}}u\right\|}_{2}^{2}={\left\|u\right\|}_{2}^{2}, , for some boundary conditions [24]. Applying the BPEG-M framework (see Algorithm A.1) to solving (2.1) leads to the following block updates:
[TABLE]
where is an extrapolation matrix that is given in (8) or (9) below, is a (scaled) majorization matrix for that is given in (7) below, , the proximal operator in (5) is given by (2), and is the characteristic function of set (i.e., equals to [math] if , and otherwise).
Proximal mapping update (3) has a single-hidden layer convolutional autoencoder architecture that consists of encoding convolution, nonlinear thresholding, and decoding convolution, where flips a filter along each dimension, and the soft-thresholding operator is defined by
[TABLE]
for , in which is the sign function. See details of deriving BPEG-M updates (3)–(5) in §A.1.4. The BPEG-M updates in (3)–(5) guarantee convergence to a critical point, when MBIR problem (2.1) satisfies some mild conditions, e.g., lower-boundedness and existence of critical points; see Assumption S.1 in §A.1.3.
The following section generalizes the BPEG-M updates in (3)–(5) and constructs the Momentum-Net architecture.
2.2 Architecture
This section establishes the INN architecture of Momentum-Net by generalizing BPEG-M updates (3)–(5) that solve (2.1). Specifically, we replace the proximal mapping in (3) with a general image refining NN , where denotes the trainable parameters. To effectively remove iteration-wise artifacts and give “best” signal estimates at each iteration, we further generalize a refining NN to iteration-wise image refining NNs , where denotes the parameters for the iteration refining NN , and is the number of Momentum-Net iterations. The iteration-wise NNs are particularly useful for reducing overfitting risks in regression, because is responsible for removing noise features only at the iteration, and thus one does not need to greatly increase dimensions of its parameter [18, 16]. In low-dose CT reconstruction, for example, the refining NNs at the early and later iterations remove streak artifacts and Gaussian-like noise, respectively [16].
Each iteration of Momentum-Net consists of 1) image refining, 2) extrapolation, and 3) MBIR modules, corresponding to the BPEG-M updates (3), (4), and (5), respectively. See the architecture of Momentum-Net in Fig. 1(a) and Algorithm 1. At the iteration, Momentum-Net performs the following three processes:
- •
Refining: The image refining module gives the “refined” image , by applying the refining NN, , to an input image at the iteration, (i.e., image estimate from the iteration). Different from existing INNs, e.g., ADMM-Net [20], PnP-ADMM [41, 23], RED [22], MoDL [30], BCD-Net [25] (see Fig. 1(b)), TNRD [21, 28], we apply -relaxation with ; see (Alg.1.1). The parameter controls the strength of inference from refining NNs, but does not affect the convergence guarantee of Momentum-Net. Proper selection of can improve MBIR accuracy (see §A.10).
- •
Extrapolation: The extrapolation module gives the extrapolated point , based on momentum terms ; see (Alg.1.2). Intuitively speaking, momentum is information from previous updates to amplify the changes in subsequent iterations. Its effectiveness has been shown in diverse optimization literature, e.g., convex optimization [48, 49] and block optimization [35, 24].
- •
MBIR: Given a refined image and a measurement vector , the MBIR module (5) applies the proximal operator to the extrapolated gradient update using a quadratic majorizer of , where is defined in (P0). Intuitively speaking, this step solves a majorized version of the following MBIR problem at the extrapolated point :
[TABLE]
and gives a reconstructed image . In Momentum-Net, we consider (non)convex differentiable MBIR cost functions with -Lipschitz continuous gradients, and a convex and closed set . For a wide range of large-scale inverse imaging problems, the majorized MBIR problem (5) has a practical closed-form solution and thus, does not require an iterative solver, depending on the properties of practically invertible majorization matrices and constraints. Examples of - combinations that give a noniterative solution for (5) include scaled identity and diagonal matrices with a box constraint and the non-negativity constraint, and matrices decomposable by unitary transforms, e.g., a circulant matrix [56, 57], with . The updated image is the input to the next Momentum-Net iteration.
The followings are details of Momentum-Net in Algorithm 1. A scaled majorization matrix is
[TABLE]
where is a symmetric positive definite majorization matrix of in the sense of -Lipschitz continuity (see Definition 1). In (7), and for convex and nonconvex (or convex and nonconvex ), respectively. We design the extrapolation matrices as follows:
[TABLE]
for some and . We update the momentum coefficients by the following formula [35, 24]:
[TABLE]
if has a sharp majorizer, i.e., has such that the corresponding bound in Definition 1 is tight, then we set , . §A.11 lists parameters of Momentum-Net, and summarizes selection guidelines or gives default values.
2.3 Relations to previous works
Several existing MBIR methods can be viewed as a special case of Momentum-Net:
- Example 3.
(MBIR model (2.1) using convolutional autoencoders that satisfy the TF condition [24]). The BPEG-M updates in (3)–(5) are special cases of the modules in Momentum-Net (Algorithm 1), with \{\mathcal{R}_{\theta^{(i+1)}}(\cdot)\!=\!\sum_{k=1}^{K}\mathrm{flip}(h_{k}^{*})\mathop{\scalebox{1.5}{\raisebox{-0.86108pt}{\ast}}}\mathcal{T}_{\beta_{k}}(h_{k}\mathop{\scalebox{1.5}{\raisebox{-0.86108pt}{\ast}}}(\cdot))\!:\!\forall i\} (i.e., single hidden-layer convolutional autoencoder [24]) and . These give a clear mathematical connection between a denoiser (3) and cost function (2.1). One can find a similar relation between a multi-hidden layer CNN and a multi-layer convolutional regularizer [24, Appx.].
- Example 4.
(INNs inspired by gradient descent method, e.g., TNRD [21, 28]). Removing extrapolation modules, i.e., setting in (Alg.1.2), and setting , Momentum-Net becomes the existing INN in [21, 28].
- Example 5.
(BCD-Net for image denoising [25]). To obtain a clean image from a noisy image corrupted by an additive white Gaussian noise (AWGN), MBIR problem (P1) considers the data-fit with the inverse covariance matrix , where is a variance of AWGN, and the box constraint with an upper bound . For this , the MBIR module (5) can use the exact majorizer and one does not need to use the extrapolation module (Alg.1.2), i.e., . Thus, Momentum-Net (with ) becomes BCD-Net.
- Example 6.
(BCD-Net for undersampled single-coil MRI [25]). To obtain an object magnetization from a k-space data obtained by undersampling (e.g., compressed sensing [58]) MRI, MBIR problem (P1) considers the data-fit with an undersampling Fourier operator (disregarding relaxation effects and considering Cartesian k-space), the inverse covariance matrix , where is a variance of complex AWGN [59], and . For this , the MBIR module (5) can use the exact majorizer that is practically invertible, where is the discrete Fourier transform and is a diagonal matrix with either [math] or (their positions correspond to sampling pattern in -space), and the extrapolation module (Alg.1.2) uses the zero extrapolation matrices . Thus, Momentum-Net (with ) becomes BCD-Net.
The following section analyzes the convergence of Momentum-Net.
2.4 Convergence analysis
In practice, INNs, i.e., “unrolled” or PnP methods using refining NNs, are trained and used with a specific number of iterations. Nevertheless, similar to optimization algorithms, studying convergence properties of INNs with [31, 23, 46] is important; in particular, it is crucial to know if a given INN tends to converge as increases. For INNs using iteration-wise refining NNs, e.g., BCD-Net [25] and proposed Momentum-Net, we expect that refiners converge, i.e., their image refining capacity converges, because information provided by data-fit function in MBIR (e.g., likelihood) reaches some “bound” after a certain number of iterations. Fig. 2 illustrates that dCNN parameters of Momentum-Net tend to converge for different applications. (The similar behavior was reported for sCNN refiners in BCD-Net [16].) Although refiners do not completely converge, in practice, one could use a refining NN at a sufficiently large iteration number, e.g., in Momentum-Net, for the later iterations.
There are two key challenges in analyzing the convergence of Momentum-Net in Algorithm 1: both challenges relate to its image refining modules (Alg.1.1). First, image refining NNs change across iterations; even if they are identical across iterations, they are not necessarily nonexpansive operators [60, 61] in practice. Second, the iteration-wise refining NNs are not necessarily proximal mapping operators, i.e., they are not written explicitly in the form of (2). This section proposes two new asymptotic definitions to overcome these challenges, and then uses those conditions to analyze convergence properties of Momentum-Net in Algorithm 1.
2.4.1 Preliminaries
To resolve the challenge of iteration-wise refining NNs and the practical difficulty in guaranteeing their non-expansiveness, we introduce the following generalized definition of the non-expansiveness [60, 61].
Definition 7** (Asymptotically nonexpansive paired operators).**
*A sequence of paired operators is asymptotically nonexpansive if there exist a summable nonnegative sequence such that111 One could replace the bound in (11) with (and summable ), and the proofs for our main arguments go through. *
[TABLE]
When and , , Definition 7 becomes the standard non-expansiveness of a mapping operator . If we replace the inequality () with the strict inequality () in (11), then we say that the sequence of paired operators is asymptotically contractive. (This stronger assumption is used to prove convergence of BCD-Net in Proposition A.5.) Definition 7 also implies that mapping operators converge to some nonexpansive operator, if the corresponding parameters converge.
Definition 7 incorporates a pairing property because Momentum-Net uses iteration-wise image refining NNs. Specifically, the pairing property helps prove convergence of Momentum-Net, by connecting image refining NNs at adjacent iterations. Furthermore, the asymptotic property in Definition 7 allows Momentum-Net to use expansive refining NNs (i.e., mapping operators having a Lipschitz constant larger than ) for some iterations, while guaranteeing convergence; see Figs. 3(a3) and 3(b3). Suppose that refining NNs are identical across iterations, i.e., , , similar to some existing INNs, e.g., PnP [23], RED [22], and other methods in §1.3. In such cases, if is expansive, Momentum-Net may diverge; this property corresponds to the limitation of existing methods described in §1.3. Momentum-Net moderates this issue by using iteration-wise refining NNs that satisfy the asymptotic paired non-expansiveness in Definition 7.
Because the sequence in (Alg.1.1) is not necessarily updated with a proximal mapping, we introduce a generalized definition of block-coordinate minimizers [53, (2.3)] for -updates:
Definition 8** (Asymptotic block-coordinate minimizer).**
The update is an asymptotic block-coordinate minimizer if there exists a summable nonnegative sequence such that
[TABLE]
Definition 8 implies that as , the updates approach a block-coordinate minimizer trajectory that satisfies (12) with . In particular, quantifies how much the update in (Alg.1.1) perturbs a block-coordinate minimizer trajectory. The bound always holds, , when one uses the proximal mapping in (3) within the BPEG-M framework. Note that while applying trained Momentum-Net, (12) is easy to examine empirically, whereas (11) is harder to check.
2.4.2 Assumptions
This section introduces and interprets the assumptions for convergence analysis of Momentum-Net in Algorithm 1:
- –
Assumption 1) In MBIR problems (P1), (non)convex is (continuously) differentiable, proper, and lower-bounded in ,222 is proper if . is lower bounded in if .
, and is convex and closed. Algorithm 1 has a fixed-point.
- –
Assumption 2) is -Lipschitz continuous with respect to (see Definition 1), where is a iteration-wise majorization matrix that satisfies with , .
- –
Assumption 3) The extrapolation matrices in (8)–(9) satisfy the following conditions:
[TABLE]
- –
Assumption 4) The sequence of paired operators is asymptotically nonexpansive with a summable sequence ; the update is an asymptotic block-coordinate minimizer with a summable sequence . The mapping functions are continuous with respect to input points and the corresponding parameters are bounded.
Assumption 1 is a slight modification of Assumption S.1 of BPEG-M, and Assumptions 2–3 are identical to Assumptions S.2–S.3 of BPEG-M; see Assumptions S.1–S.3 in §A.1.3. The extrapolation matrix designs (8) and (9) satisfy conditions (– ‣ 2.4.2) and (– ‣ 2.4.2) in Assumption 3, respectively.
We provide empirical justifications for the first two conditions in Assumption 4. First, Figs. 3(a2) and A.1(a2) illustrate that paired refining NNs of Momentum-Net appear to be asymptotically nonexpansive in an application that has mild condition number variations across training data-fit majorization matrices. Figs. 3(a3), 3(b3), A.1(a3), and A.1(b3) illustrate for different applications that refining NNs become nonexpansive: their Lipschitz constants at the first several iterations are larger than , and their Lipschitz constants in later iterations become less than . Alternatively, the asymptotic non-expansiveness of paired operators can be satisfied by a stronger assumption that the sequence converges to some nonexpansive operator. (Fig. 2 illustrates that dCNN parameters of Momentum-Net appear to converge.)
Figs. 3(a3), 3(b3), A.1(a3), and A.1(b3) illustrate for different applications that the -updates are asymptotic block-coordinate minimizers. Lemma A.4 and §A.3 in the appendices provide a probabilistic justification for the asymptotic block-coordinate minimizer condition.
2.4.3 Main convergence results
This section analyzes fixed-point and critical point convergence of Momentum-Net in Algorithm 1, under the assumptions in the previous section. We first show that differences between two consecutive iterates generated by Momentum-Net converge to zero:
Proposition 9** (Convergence properties).**
Under Assumptions 1–4, let be the sequence generated by Algorithm 1. Then, the sequence satisfies
[TABLE]
and hence {\left\|\left[\begin{array}[]{c}x^{(i+1)}\\ z^{(i+1)}\end{array}\right]-\left[\begin{array}[]{c}x^{(i)}\\ z^{(i)}\end{array}\right]\right\|}_{2}\rightarrow 0.
Proof.
See §A.4 in the appendices. ∎
Using Proposition 9, our main theorem provides that any limit points of the sequence generated by Momentum-Net satisfy critical point and fixed-point conditions:
Theorem 10** (A limit point satisfies both critical point and fixed-point conditions).**
Under Assumptions 1–4 above, let be the sequence generated by Algorithm 1. Consider either a fixed majorization matrix with general structure, i.e., for , or a sequence of diagonal majorization matrices, i.e., . Then, any limit point of satisfies both the critical point condition:
[TABLE]
where is a limit point of , and the fixed-point condition:
[TABLE]
where , denotes performing the updates in Algorithm 1, and and is a limit point of and , respectively.
Proof.
See §A.5 in the appendices. ∎
Observe that, if or is an interior point of , (16) reduces to the first-order optimality condition , where denotes the limiting subdifferential of at . With additional isolation and boundedness assumptions for the points satisfying (16) and (17), we obtain whole sequence guarantees:
Corollary 11** (Whole sequence convergence).**
Consider the construction in Theorem 10. Let be the set of points satisfying the critical point condition in (16) and the fixed-point condition in (17). If is bounded, then , where denotes the distance from to , for any point and any subset . If contains uniformly isolated points, i.e., there exists such that for any distinct points , then converges to a point in .
Proof.
See §A.6 in the appendices. ∎
The boundedness assumption for in Corollary 11 is standard in block-wise optimization, e.g., [62, 53, 55, 35, 24]. The assumption can be satisfied if the set is bounded (e.g., box constraints), one chooses appropriate regularization parameters in Algorithm 1 [55, 35, 24], the function is coercive [62], or the level set is bounded [53]. However, for general , it is hard to verify the isolation condition for the points in in practice. Instead, one may use Kurdyka-Łojasiewicz property [62, 53] to analyze the whole sequence convergence with some appropriate modifications.
For simplicity, we focused our discussion to noniterative MBIR module (5). However, Momentum-Net practically converges with any proximable MRIR function (5) that may need an iterative solver, if sufficient inner iterations are used. To maximize the computational benefit of Momentum-Net, one needs to make sure that majorized MBIR function (5) is better proximable over its original form (P1).
2.5 Benefits of Momentum-Net
Momentum-Net has several benefits over existing INNs:
- –
Benefits from refining module: The image refining module (Alg.1.1) can use iteration-wise image refining NNs : those are particularly useful to reduce overfitting risks by reducing dimensions of their parameters at each iteration [18, 16, 19]. Iteration-wise refining NNs require less memory for training, compared to methods that use a single refining NN for all iterations, e.g., [63]. Different from the existing methods mentioned in §1.3, Momentum-Net does not require (firmly) nonexpansive mapping operators to guarantee convergence. Instead, in (Alg.1.1) assumes a generalized notion of the (firm) non-expansiveness condition assumed for convergence of the existing methods that use identical refining NNs across iterations, including PnP [20, 39, 40, 41, 23, 46], RED [22, 43], etc. The generalized concept is the first practical condition to guarantee convergence of INNs using iteration-wise refining NNs; see Definition 7.
- –
Benefits from extrapolation module: The extrapolation module (Alg.1.2) uses the momentum terms that accelerate the convergence of Momentum-Net. In particular, compared to the existing gradient-descent-inspired INNs, e.g., TNRD [21, 28], Momentum-Net converges faster. (Note that the way the authors of [43] used momentum is less conventional. The corresponding method, RED-APG [43, Alg. 6], still can require multiple inner iterations to solve its quadratic MBIR problem, similar to BCD-Net-type methods.)
- –
Benefits from MBIR module: The MBIR module (5) does not require multiple inner iterations for a wide range of imaging problems and has both theoretical and practical benefits. Note first that convergence analysis of INNs (including Momentum-Net) assumes that their MBIR operators are noniterative. In other words, related convergence theory (e.g., Proposition A.5) is inapplicable if iterative methods, particularly with insufficient number of iterations, are applied to MBIR modules. Different from the existing BCD-Net-type methods [25, 20, 39, 40, 41, 22, 30, 43, 23] that can require iterative solvers for their MBIR modules, MBIR module (5) of Momentum-Net can have practical close-form solution (see examples in §2.2), and its corresponding convergence analysis (see §2.4) can hold stably for a wide range of imaging applications. Second, combined with extrapolation module (Alg.1.2), noniterative MBIR modules (5) lead to faster MBIR, compared to the existing BCD-Net-type methods that can require multiple inner iterations for their MBIR modules for convergence. Third, Momentum-Net guarantees convergence even for nonconvex MBIR cost function or nonconvex data-fit of which the gradient is -Lipschitz continuous (see Definition 1), while existing INNs overlooked nonconvex or .
Furthermore, §A.7 analyzes the sequence convergence of BCD-Net [25], and describes the convergence benefits of Momentum-Net over BCD-Net.
3 Training INNs
This section describes training of all the INNs compared in this paper.
3.1 Architecture of refining NNs and their training
For all INNs in this paper, we train the refining NN at each iteration to remove artifacts from the input image that is fed from the previous iteration. For the iteration NN, we first consider the following sCNN architecture, residual single-hidden layer convolutional autoencoder:
[TABLE]
where is the parameter set of the image refining NN, is a set of decoding and encoding filters of size , is a set of thresholding values, and is the soft-thresholding operator with parameter defined in (6), for . We use the exponential function to prevent the thresholding parameters from becoming negative during training. We observed that the residual convolutional autoencoder in (18) gives better results compared to the convolutional autoencoder, i.e., (18) without the second term [25, 18]. This corresponds to the empirical result in [64, 65] that having skip connections (e.g., the second term in (18)) can improve generalization. The sequence of paired sCNN refiners (18) can satisfy the asymptotic non-expansiveness, if its convergent refiner satisfies that
[TABLE]
where is the largest eigenvalue of a matrix, , , are limit point filters, and is the Kronecker delta filter of size .
For dCNN refiners, we use the following residual multi-hidden layer CNN architecture, a simplified DnCNN [4] using fewer layers, no pooling, and no batch normalization [46] (we drop superscript indices for simplicity):
[TABLE]
for and , where is the parameter set of each refining NN, is the number of feature maps, is the number of layers, is a set of filters at the first and last dCNN layer, is a set of filters for remaining dCNN layer, and the rectified linear unit activation function is defined by .
The training process of Momentum-Net requires high-quality training images, , training measurements simulated via imaging physics, , and data-fits and the corresponding majorization matrices . Different from [25, 16, 66] that train convolutional autoencoders from the patch perspective, we train the image refining NNs in (18)–(19) from the convolution perspective (that does not store many overlapping patches, e.g., see [24]). From training pairs , where is a set of reconstructed images at the Momentum-Net iteration, we train the iteration image refining NN in (18) by solving the following optimization problem:
[TABLE]
where is given as in (18), for (see some related properties in §A.8). We solve the training optimization problems (P2) by mini-batch stochastic optimization with the subdifferentials computed by the PyTorch Autograd package.
3.2 Regularization parameter selection based on “spectral spread”
When majorization matrices of training data-fits have similar spectral properties, e.g., condition numbers, the regularization parameter in (P1) is trainable by substituting (Alg.1.1) into (5) and modifying the training cost (P2). However, the condition numbers of data-fit majorizers can greatly differ due a variety of imaging geometries or image formation systems, or noise levels in training measurements, etc. See such examples in §4.1–4.2.
To train Momentum-Net with diverse training data-fits, we propose a parameter selection scheme based on the “spectral spread” of their majorization matrices . For simplicity, consider majorization matrices of the form , where the factor is selected by (7) and is a symmetric positive semidefinite majorization matrix for , . We select the regularization parameter for the training sample as
[TABLE]
where the spectral spread of a symmetric positive definite matrix is defined by for , and is the smallest eigenvalue of a matrix. For the training sample, a tunable factor controls in (20) according to , . The proposed parameter selection scheme also applies to testing Momentum-Net, based on the tuned factor in its training. We observed that the proposed parameter selection scheme (20) gives better MBIR accuracy than the condition number based selection scheme that is similarly used in selecting ADMM parameters [17] (for the two applications in §4). One may further apply this scheme to iteration-wise majorization matrices and select iteration-wise regularization parameters accordingly. For comparing different INNs, we apply (20) to all INNs.
4 Experimental results and discussion
We investigated two extreme imaging applications: sparse-view CT and LF photography using a focal stack. In particular, these two applications lack a practical closed-form solution for the MBIR modules of BCD-Net and ADMM-Net [20], e.g., solving (Alg.2.2). For these applications, we compared the performances of the following five INNs: BCD-Net [25] (i.e., generalization of RED [22] and MoDL [30]), ADMM-Net [20], i.e., PnP-ADMM [41, 23] using iteration-wise refining NNs, Momentum-Net without extrapolation (i.e., generalization of TNRD [21, 28]), PDS-Net, i.e., PnP-PDS [50] using iteration-wise refining NNs, and the proposed Momentum-Net using extrapolation.
4.1 Experimental setup: Imaging
4.1.1 Sparse-view CT
To reconstruct a linear attenuation coefficient image from post-log sinogram in sparse-view CT, the MBIR problem (P1) considers a data-fit and the non-negativity constraint , where is an undersampled CT system matrix, is a diagonal weighting matrix with elements based on a Poisson-Gaussian model [67, 17] for the pre-log raw measurements with electronic readout noise variance .
We simulated 2D sparse-view sinograms of size – ‘detectors or rays’ ‘regularly spaced projection views or angles’, where is the number of full views – with GE LightSpeed fan-beam geometry corresponding to a monoenergetic source with incident photons per ray and no background events, and electronic noise variance . We avoided an inverse crime in imaging simulation and reconstructed images of size with a coarser grid mm; see details in [37, §V-A2].
4.1.2 LF photography using a focal stack
To reconstruct a LF that consists of sub-aperture images from focal stack measurements that are collected by photosensors, the MBIR problem (P1) considers a data-fit and a box constraint with (or without rescaling), where is a system matrix of LF imaging system using a focal stack that is constructed blockwise with different convolution matrices [47, 68], is a transparency coefficient for the detector, and is the size of sub-aperture images, , .333 Traditionally, one obtains focal stacks by physically moving imaging sensors and taking separate exposures across time. Transparent photodetector arrays [47, 69] allow one to collect focal stack data in a single exposure, making a practical LF camera using a focal stack. If some photodetectors are not perfectly transparent, one can use , for some .
In general, a LF photography system using a focal stack is extremely under-determined, because .
To avoid an inverse crime, our imaging simulation used higher-resolution synthetic LF dataset [70] (we converted the original RGB sub-aperture images to grayscale ones by the “rgb2gray.m” function in MATLAB, for simplicity and smaller memory requirements in training). We simulated focal stack images of size with dB AWGN that models electronic noise at sensors, and setting transparency coefficients as , for . The sensor positions were chosen such that five sensors focus at equally spaced depths; specifically, the closest sensor to scenes and farthest sensor from scenes focus at two different depths that correspond to ‘’ and ‘’, respectively, where and are the approximate maximum and minimum disparity values specified in [70]. We reconstructed 4D LFs that consist of sub-aperture images of size , with a coarser grid mm.
4.2 Experimental setup: INNs
4.2.1 Parameters of INNs
The parameters for the INNs compared in sparse-view CT experiments were defined as follows. We considered two BCD-Nets (see Algorithm 2): for one BCD-Net, we applied the APG method [49] with inner iterations to (Alg.2.2), and set ; for the other BCD-Net, we applied the APG method with inner iterations to (Alg.2.2), and set . For ADMM-Net, we used the identical configurations as BCD-Net and set the ADMM penalty parameter to in (P1), similar to [16]. For Momentum-Net without extrapolation, we chose and . For the proposed Momentum-Net, we chose and . For PDS-Net, we set the first step size to and the second step size to , per [50]. For performance comparisons between different INNs, all the INNs used sCNN refiners (18) with to avoid the overfitting/hallucination risks. For Momentum-Net using dCNN refiners, we chose layer dCNN (19) using filters and feature maps, following [46]. (The chosen parameters gave lower RMSE values than , for identical regularization parameters.) For comparing different MBIR methods, Momentum used extrapolation, i.e., (Alg.1.2) with (8) and (10), and for (18). We designed the majorization matrices as , using Lemma A.7 ( and have nonnegative entries) and setting by (7). We set an initial point of INNs, , to filtered-back projection (FBP) using a Hanning window. The regularization parameters of all INNs were selected by the scheme in §3.2 with . (This factor was estimated from the carefully chosen regularization parameter for sparse-view CT MBIR experiments using learned convolutional regularizers in [24].)
The parameters for the INNs compared in experiments of LF photography using a focal stack were defined as follows. We considered two BCD-Nets and two ADMM-Nets with the identical parameters listed above. For Momentum-Net without extrapolation and the proposed Momentum-Net, we set and . For PDS-Net, we used the identical parameter setup described above. For performance comparisons between different INNs, all the INNs used sCNN refiners (18) with (to avoid the overfitting risks) in the epipolar domain. For Momentum-Net using dCNN refiners, we chose layer dCNN (19) using filters and feature maps. (The chosen parameters gave most accurate performances over the following setups, , given the identical regularization parameters.) To generate in (Alg.1.1), we applied a sCNN (18) with or a dCNN (19) with to two sets of horizontal and vertical epipolar plane images, and took the average of two LFs that were permuted back from refined horizontal and vertical epipolar plane image sets, .444 Epipolar images are 2D slices of a 4D LF , where and are spatial and angular coordinates, respectively. Specifically, each horizontal epipolar plane image are obtained by fixing and , and varying and ; and each vertical epipolar image are obtained by fixing and , and varying and .
We designed the majorization matrices as , using Lemma A.7 and setting by (7). We set an initial point of INNs, , to rescaled in the interval (i.e., dividing by its max value). The regularization parameters (i.e., in BCD-Net/Momentum-Net, the ADMM penalty parameter in ADMM-Net, and the first step size in PDS-Net) were selected by the proposed scheme in §3.2 with . (We tuned the factor to achieve the best performances).
For different combinations of INNs and sCNN refiner (18)/dCNN refiner (19), we use the following naming convention: ‘the INN name’-‘sCNN’ or ‘dCNN’.
4.2.2 Training INNs
For sparse-view CT experiments, we trained all the INNs from the chest CT dataset with ; we constructed the dataset by using XCAT phantom slices [71]. The CT experiment has mild data-fit variations across training samples: the standard deviation of the condition numbers () of is . For experiments of LF photography using a focal stack, we trained all the INNs from the LF photography dataset with and two sets of ground truth epipolar images, ; we constructed the dataset by excluding four unrealistic “stratified” scenes from the original LF dataset in [70] that consists of LFs with diverse scene parameter and camera settings. The LF experiment has large data-fit variations across training samples: the standard deviation of the condition numbers of is .
In training INNs for both the applications, if not specified, we used identical training setups. At each iteration of INNs, we solved (P2) with the mini-batch version of Adam [72] and trained iteration-wise sCNNs (18) or dCNNs (19). We selected the batch size and the number of epochs as follows: for sparse-view CT reconstruction, we chose them as & , and & for sCNN and dCNN refiners, respectively; for LF photography using a focal stack, we chose them as & , and & , for sCNN and dCNN refiners, respectively. We chose the learning rates for filters in sCNNs and dCNNs, and thresholding values in sCNNs (18), as and , respectively; we reduced the learning rates by % every epochs. At the first iteration, we initialized filter coefficients with Kaiming uniform initialization [73]; in the later iteration, i.e., at the INN iteration, for , we initialized filter coefficients from those learned from the previous iteration, i.e., iteration (this also applies to initializing thresholding values).
4.2.3 Testing trained INNs
In sparse-view CT reconstruction experiments, we tested trained INNs to two samples where ground truth images and the corresponding inverse covariance matrices (i.e., in §4.1.1) sufficiently differ from those in training samples (i.e., they are a few cm away from training images). We evaluated the reconstruction quality by the most conventional error metric in CT application, RMSE (in HU), in a region of interest (ROI), where RMSE and HU stand for root-mean-square error and (modified) Hounsfield unit, respectively, and the ROI was a circular region that includes all the phantom tissues. The RMSE is defined by , where is a reconstructed image, is a ground truth image, and is the number of pixels in a ROI. In addition, we compared the trained Momentum-Net (using extrapolation) to a standard MBIR method using a hand-crafted EP regularizer, and an MBIR model using a learned convolutional regularizer [24, 37] which is the state-of-the-art MBIR method within an unsupervised learning setup. We finely tuned their regularization parameters to achieve the lowest RMSE. See details of these two MBIR models in §A.9.2.
In experiments of LF photography using a focal stack, we tested trained INNs to three samples of which scene parameter and camera settings are different from those in training samples (all training and testing samples have different camera and scene parameters). We evaluated the reconstruction quality by the most conventional error metric in LF photography application, PSNR (in dB), where PSNR stands for peak signal-to-noise. In addition, we compared the trained Momentum-Net (using extrapolation) to MBIR method using the state-of-the-art non-trained regularizer, 4D EP introduced in [47]. (The low-rank plus sparse tensor decomposition model [68, 74] failed when inverse crimes and measurement noise are considered.) We finely tuned its regularization parameter to achieve the lowest RMSE values. See details of this MBIR model in §A.9.3. We further investigated impacts of the LF MBIR quality on a higher-level depth estimation application, by applying the robust Spinning Parallelogram Operator (SPO) depth estimation method [75] to reconstructed LFs.
For comparing Momentum-Net with PDS-Net, we measured quality of refined images, in (Alg.1.1), because PDS-Net is a hard-refiner.
The imaging simulation and reconstruction experiments were based on the Michigan image reconstruction toolbox [76], and training INNs, i.e., solving (P2), was based on PyTorch (for sparse-view CT, we used ver. 1.2.0; for LF photography using a focal stack, we used ver. 0.3.1). For sparse-view CT, single-precision MATLAB and PyTorch implementations were tested on GHz Intel Core i7 CPU with GB RAM, and 1405 MHz Nvidia Titan Xp GPU with 12 GB RAM, respectively. For LF photography using a focal stack, they were tested on GHz AMD Threadripper 1920X CPU with GB RAM, and MHz Nvidia GTX 1080 Ti GPU with GB RAM, respectively.
4.3 Comparisons between different INNs
First, compare sCNN results in Figs. 4–5 and Figs. 6–7, for sparse-view CT and LF photography using a focal stack, respectively. For both applications, the proposed Momentum-Net using extrapolation significantly improves MBIR speed and accuracy, compared to the existing soft-refining INNs, [30, 22, 23, 21, 28] that correspond to BCD-Net [25] or Momentum-Net using no extrapolation, and ADMM-Net [20, 41, 23], and the existing hard-refining INN PDS-Net [50]. (Note that BCD-Net and Momentum-Net require slightly less computational complexity per INN iteration, compared to ADMM-Net and PDS-Net, respectively, due to having fewer modules.) Fig. 5 shows that to reach the HU RMSE value in sparse-view CT reconstruction, the proposed Momentum-Net decreases MBIR time by % and %, compared to Momentum-Net without extrapolation and BCD-Net using three inner iterations, respectively. Fig. 7 shows that to reach the dB PSNR value in LF reconstruction from a focal stack, the proposed Momentum-Net decreases MBIR time by % and %, compared to Momentum-Net without extrapolation and BCD-Net using three inner iterations, respectively. In addition, Figs. 5 and 7 show that using extrapolation, i.e., (Alg.1.2) with (8)–(10), improves the performance of Momentum-Net versus iterations.
We conjecture that the larger performance gap between soft-refiner Momentum-Net and hard-refiner PDS-Net, in Fig. 4(a) compared to Fig. 6(a), is because the LF problem needs stronger regularization, i.e., a smaller tuned factor in (20), than the CT problem. Similarly, comparing Fig. 4(b) to Fig. 6(b) shows that the LF problem has small performance gaps between BCD-Net and ADMM-Net.
For both the applications, using dCNN refiners (19) instead of sCNN refiners (18) has a negligible effect on total run time of Momentum-Net, because reconstruction time of MBIR modules (5) (in CPUs) dominates inference time of image refining modules (Alg.1.1) (in GPUs). Compare results between Momentum-Net-sCNN and -dCNN in Figs. 5 & 7 and Tables A.1 & A.2.
4.4 Comparisons between different MBIR methods
In sparse-view CT using % of the full projection views, Fig. 8(b)–(e) and Table A.1(b)–(f) show that the proposed Momentum-Net achieves significantly better reconstruction quality compared to the conventional EP MBIR method and the state-of-the-art MBIR method within an unsupervised learning setup, MBIR model using a learned convolutional regularizer [24, 37]. In particular, Momentum-Net recovers both low- and high-contrast regions (e.g., soft tissues and bones, respectively) more accurately than MBIR using a learned convolutional regularizer; see Fig. 8(c)–(e). In addition, when their shallow convolutional autoencoders need identical computational complexities, Momentum-Net achieves much faster MBIR compared to MBIR using a learned convolutional regularizer; see Table A.1(c)–(d).
In LF photography using five focal sensors, regardless of scene parameters and camera settings, Momentum-Net consistently achieves significantly more accurate image recovery, compared to MBIR model using the state-of-the-art non-trained regularizer, 4D EP [47]. The effectiveness of Momentum-Net is more evident for a scene with less fine details. See Fig. 9(b)–(d) and Table A.2(b)–(d). Regardless of the scene distances from LF imaging systems, the reconstructed LFs by Momentum-Net significantly improve the depth estimation accuracy over those reconstructed by the state-of-the-art non-trained regularizer, 4D EP [47]. See Fig. 10(c)–(e) and Table A.3(c)–(e).
In general, Momentum-Net needs more computations per iteration than EP MBIR, because its refining NNs use more and larger filters than the small finite-difference filters in EP MBIR, and EP MBIR algorithms can be often further accelerated by gradient approximations, e.g., ordered-subsets methods [77, 78].
5 Conclusions
Developing rapidly converging INNs is important, because 1) it leads to fast MBIR by reducing the computational complexity in calculating data-fit gradients or applying refining NNs, and 2) training INNs with many iterations requires long training time or it is challenging when refining NNs are fixed across INN iterations. The proposed Momentum-Net framework is applicable for a wide range of inverse problems, while achieving fast and convergent MBIR. To achieve fast MBIR, Momentum-Net uses momentum in extrapolation modules, and noniterative MBIR modules at each iteration via majorizers. For sparse-view CT and LF photography using a focal stack, Momentum-Net achieves faster and more accurate MBIR compared to the existing soft-refining INNs, [30, 22, 23, 21, 28] that correspond to BCD-Net [25] or Momentum-Net using no extrapolation, and ADMM-Net [20, 41, 23], and the existing hard-refining INN PDS-Net [50]. When an application needs strong regularization strength, e.g., LF photography using limited detectors, using dCNN refiners with moderate depth significantly improves the MBIR accuracy of Momentum-Net compared to sCNNs, only marginally increasing total MBIR time. In addition, Momentum-Net guarantees convergence to a fixed-point for general differentiable (non)convex MBIR functions (or data-fit terms) and convex feasible sets, under some mild conditions and two asymptotic conditions. The proposed regularization parameter selection scheme uses the “spectral spread” of majorization matrices, and is useful to consider data-fit variations across training/testing samples.
There are a number of avenues for future work. First, we expect to further improve performances of Momentum-Net (e.g., MBIR time and accuracy) by using sharper majorizer designs. Second, we expect to further reduce MBIR time of Momentum-Net with the stochastic gradient perspective (e.g., ordered subset [77, 78]). On the regularization parameter selection side, our future work is learning the factor in (20) from datasets while training refining NNs.
A.1 BPEG-M: Review
This section explains block multi-(non)convex optimization problems, and summarizes the state-of-the-art method for block multi-(non)convex optimization, BPEG-M [Chun&Fessler:18TIP:supp, Chun&Fessler:20TIP:supp], along with its convergence guarantees.
A.1.1 Block multi-(non)convex optimization
In a block optimization problem, the variables of the underlying optimization problem are treated either as a single block or multiple disjoint blocks. In block multi-(non)convex optimization, we consider the following problem:
[TABLE]
where variable is decomposed into blocks (), is assumed to be (continuously) differentiable, but functions are not necessarily differentiable. The function can incorporate the constraint , by allowing ’s to be extended-valued, e.g., if , for . It is standard to assume that both and are proper and closed, and the sets are closed. We consider either that (A.1) has block-wise convexity (but (A.1) is jointly nonconvex in general) [Xu&Yin:13SIAM:supp, Chun&Fessler:18TIP:supp] or that , , or are not necessarily convex [Xu&Yin:17JSC:supp, Chun&Fessler:20TIP:supp]. Importantly, can include (non)convex and nonsmooth (quasi-)norm, . The next section introduces our optimization framework that solves (A.1).
The following sections review BPEG-M [Chun&Fessler:18TIP:supp, Chun&Fessler:20TIP:supp], the state-of-the-art optimization framework for solving block multi-(non)convex problems, when used with sufficiently sharp majorizers. BPEG-M uses block-wise extrapolation, majorization, and proximal mapping. By using a more general Lipschitz continuity (see Definition 1) for block-wise gradients, BPEG-M is particularly useful for rapidly calculating majorizers involved with large-scale problems, and successfully applied to some large-scale machine learning and computational imaging problems; see [Chun&Fessler:18TIP:supp, Chun&Fessler:20TIP:supp, Chun&Fessler:18Asilomar:supp] and references therein.
A.1.2 BPEG-M
This section summarizes the BPEG-M framework. Using Definition 1 and Lemma 2, the proposed method, BPEG-M, is given as follows. To solve (A.1), we minimize majorizers of cyclically over each block , while fixing the remaining blocks at their previously updated variables. Let be the value of after its update, and define
[TABLE]
for all . At the block of the iteration, we apply Lemma 2 to functional with a -Lipschitz continuous gradient at the extrapolated point , and minimize a majorized function. In other words, we consider the updates
[TABLE]
where
[TABLE]
the proximal operator is defined by (2), is the block-partial gradient of at , a scaled majorization matrix is given by
[TABLE]
and is a symmetric positive definite majorization matrix of . In (A.3), the matrix is an extrapolation matrix that accelerates convergence in solving block multi-convex problems [Chun&Fessler:18TIP:supp]. We design it to satisfy conditions (A.5) or (A.6) below. In (A.4), and , for block multi-convex and block multi-nonconvex problems, respectively.
For some having sharp majorizers, we expect that extrapolation (A.3) has no benefits in accelerating convergence, and use . Other than the blocks having sharp majorizers, one can apply some increasing momentum coefficient formula [Nesterov:13MP:supp, Beck&Teboulle:09SIAM:supp] to the corresponding extrapolation matrices. The choice in [Chun&Fessler:18TIP:supp, Chun&Fessler:20TIP:supp, Xu&Yin:13SIAM:supp] accelerated BPEG-M for some machine learning and data science applications. Algorithm A.1 summarizes these updates.
A.1.3 Convergence results
This section summarizes convergence results of Algorithm A.1 under the following assumptions:
- –
Assumption S.1) In (A.1), is proper and lower bounded in . In addition,
- for block multi-convex (A.1), is differentiable and (A.1) has a Nash point or block-coordinate minimizerA.1A.1A.1 Given a feasible set , a point is a critical point (or stationary point) of if the directional derivative for any feasible direction at . If is an interior point of , then the condition is equivalent to . (see its definition in [Xu&Yin:13SIAM:supp, (2.3)–(2.4)]);
- for block multi-nonconvex (A.1), is continuously differentiable, is lower semicontinuousA.2A.2A.2 is lower semicontinuous at point if . , , and (A.1) has a critical point that satisfies .
- –
Assumption S.2) is -Lipschitz continuous with respect to , i.e.,
[TABLE]
for , where is a bounded majorization matrix.
- –
Assumption S.3) The extrapolation matrices satisfy that
[TABLE]
with , .
Theorem A.1** (Block multi-convex (A.1): A limit point is a Nash point [Chun&Fessler:18TIP:supp]).**
Under Assumptions S.1–S.3, let be the sequence generated by Algorithm A.1. Then any limit point of is a Nash point of (A.1).
Theorem A.2** (Block multi-nonconvex (A.1): A limit point is a critical point [Chun&Fessler:20TIP:supp]).**
Under Assumptions S.1–S.3, let be the sequence generated by Algorithm A.1. Then any limit point of is a critical point of (A.1).
- Remark A.3.
Theorems A.1–A.2 imply that, if there exists a critical point for (A.1), i.e., , then any limit point of is a critical point. One can further show global convergence under some conditions: if is bounded and the critical points are isolated, then converges to a critical point [Chun&Fessler:18TIP:supp, Rem. 3.4], [Xu&Yin:13SIAM:supp, Cor. 2.4].
A.1.4 Application of BPEG-M to solving block multi-(non)convex problem (2.1)
For update (3), we do not use extrapolation, i.e., (A.3), since the corresponding majorization matrices are sharp, so one obtains tight majorization bounds in Lemma 2. See, for example, [Chun&Fessler:20TIP:supp, §V-B]. For updates (3) and (5), we rewrite \sum_{k=1}^{K}\|h_{k}\mathop{\scalebox{1.5}{\raisebox{-0.86108pt}{\ast}}}x-\zeta_{k}\|_{2}^{2} as \|x-\sum_{k=1}^{K}\mathrm{flip}(h_{k}^{*})\mathop{\scalebox{1.5}{\raisebox{-0.86108pt}{\ast}}}\zeta_{k}\|_{2}^{2} by using the TF condition in §2.1 [Chun&Fessler:20TIP:supp, §VI], [Chun&Fessler:18Asilomar:supp].
A.2 Empirical measures related to the convergence of Momentum-Net using sCNN refiners
This section provides empirical measures related to Assumption 4 for Momentum-Net using single-hidden layer autoencoders (18); see Fig. A.1 below. We estimated the sequence in Definition 7, the sequence in Definition 8, and the Lipschitz constants of refining NNs , based on a hundred sets of randomly selected training samples related with the corresponding bounds of the measures, e.g., and in (11) are training input to and in (Alg.1.1), respectively.
A.3 Probabilistic justification for the asymptotic block-coordinate minimizer condition in Assumption 4
This section introduces a useful result for an asymptotic block-coordinate minimizer : the following lemma provides a probabilistic bound for in (12), given a subgaussian vector with independent and zero-mean entries.
Lemma A.4** (Probabilistic bounds for ).**
Assume that is a zero-mean subgaussian vector of which entries are independent and zero-mean subgaussian variables. Then, each bound in (12) holds with probability at least
[TABLE]
where is a subgaussian parameter for , and a random variable is subgaussian with parameter if for .
Proof.
First, observe that
[TABLE]
where the inequality (A.7) holds by via (Alg.1.1). We now obtain a probablistic bound for the third quantity in (A.8) via a concentration inequality. The concentration inequality on the sum of independent zero-mean subgaussian variables (e.g., [Foucart&Rauhut:book:supp, Thm. 7.27]) yields that for any
[TABLE]
where is given as in Lemma A.4. Applying the result (A.9) with to the bound (A.8) completes the proofs. ∎
Lemma A.4 implies that, given sufficiently large , or sufficiently small (e.g., variance for a Gaussian random vector ) or , bound (12) is satisfied with high probability, for each . In particular, can be large for the first several iterations; if paired operators in (Alg.1.1) map their input images to similar output images (e.g., the trained NNs and have good refining capabilities for and ), then is small; if the regularization parameter in (5) is sufficiently large, then is small.
A.4 Proofs of Proposition 9
First, we show that for convex and nonconvex cases.
- –
Convex case: Using Assumption 2 and for the convex case via (7), we obtain the following results for any :
[TABLE]
where the inequality (A.10) uses the condition (12) in Assumption 4, the inequality (A.11) is obtained by using the results in [Chun&Fessler:18TIP:supp, Lem. S.1], the equality (A.12) uses the extrapolation formula (Alg.1.2) and the symmetry of , the inequality (A.13) holds by Assumption 3.
Summing the inequality of in (A.13) over , we obtain
[TABLE]
where the inequality (A.14) holds by Assumption 2. Due to the lower boundedness of in Assumption 1 and the summability of in Assumption 4, taking gives
[TABLE]
- –
Nonconvex case: Using Assumption 2, we obtain the following results without assuming that is convex:
[TABLE]
where the inequality (A.16) uses the condition (12) in Assumption 4, the inequality (A.17) use the results in [Chun&Fessler:20TIP:supp, §S.3], the equality (A.18) holds by (Alg.1.2), the inequality (A.19) is obtained by Assumption 3.
Summing the inequality of in (A.19) over , we obtain
[TABLE]
where we follow the arguments in obtaining (A.14) above. Again, using the lower boundedness of and the summability of , taking gives the result (A.15) for nonconvex .
Second, we show that . Observe
[TABLE]
where the first equality uses the image mapping formula in (Alg.1.1), the first inequality holds by applying Jensen’s inequality to the (convex) squared -norm, the second inequality is obtained by using the asymptotically non-expansiveness of the paired operators in Assumption 4. Summing the inequality of in (A.20) over , we obtain
[TABLE]
where we used as given in Algorithm 1. By taking in (A.21), using result (A.15), and the summability of the sequence , we obtain
[TABLE]
Combining the results in (A.15) and (A.22) completes the proofs.
A.5 Proofs of Theorem 10
Let be a limit point of and be the subsequence converging to . Let be a limit point of and be the subsequence converging to . The closedness of implies that . Using the results in Proposition 9, and also converge to and , respectively. Taking another subsequence if necessary, the subsequence converges to some , since is bounded by Assumption 2. The subsequences converge to some , since , , and is bounded via Assumption 4.
Next, we show that the convex proximal minimization (A.23) below is continuous in the sense that the output point continuously depends on the input points and , and majorization matrix :
[TABLE]
where the proximal mapping operator is given as in (2). We consider the two cases of majorization matrices given in Theorem 10:
- –
For a sequence of diagonal majorization matrices, i.e., , one can obtain the continuity of the convex proximal minimization (A.23) with respect to , , and , by extending the existing results in [Rockafellar&Wets:book:supp, Thm. 2.26], [Rockafellar:76SIAMCO:supp] with the separability of (A.23) to element-wise optimization problems.
- –
For a fixed general majorization matrix, i.e., , , we obtain that the convex proximal minimization (A.24) below is continuous with respect to the input points and :
[TABLE]
where is the subdifferential of at and denotes the identity operator, and the proximal mapping of relative to is uniquely determined by the resolvent of the operator in (A.25).
First, we obtain that the operator is monotone. For a convex extended-valued function , observe that is a monotone operator:
[TABLE]
where the equality uses the variable change , a chain rule of the subdifferential of a composition of a convex extended-valued function and an affine mapping [Mordukhovich&Nam:17Opt:supp, §7], and the symmetry of , and the inequality holds because the subdifferential of convex extended-valued function is a monotone operator [Ryu&Boyd:16ACM:supp, §4.2]. Because characteristic function of a convex set is extended-valued function, the result in (A.26) implies that the operator is monotone. Second, note that the resolvent of a monotone operator (with a parameter ), i.e., in (A.25), is nonexpansive [Rockafellar&Wets:book:supp, §6] and thus continuous. We now obtain that the convex proximal minimization (A.24) is continuous with respect to the input points and , because the proximal mapping operator in (A.25), the affine mapping , and are continuous with respect to their input points.
For the two cases above, using the fact that , , , and (or for the case) as , (A.23) becomes
[TABLE]
Thus, satisfies the first-order optimality condition of :
[TABLE]
and this completes the proof of the first result.
Next, note that the result in Proposition 9 imply
[TABLE]
Additionally, note that a function is continuous. To see this, observe that the convex proximal mapping in (5) is continuous (see the obtained results above), and is continuous (see Assumption 4). Combining (A.28), the convergence of , and the continuity of , we obtain , and this completes the proofs of the second result.
A.6 Proofs of Corollary 11
To prove the first result, we use proof by contradiction. Suppose that . Then there exists and a subsequence such that , . However, the boundedness assumption of in Corollary 11 implies that there must exist a limit point via Theorem 10. This is a contradiction, and gives the first result (via the result in Proposition 9). Under the isolation point assumption in Corollary 11, using the obtained results, (via Proposition 9) and , and the following the proofs in [Xu&Yin:13SIAM:supp, Cor. 2.4], we obtain the second result.
A.7 Momentum-Net vs. BCD-Net
This section compares the convergence properties of Momentum-Net (Algorithm 1) and BCD-Net (Algorithm 2). We first show that for convex and , the sequence of reconstructed images generated by BCD-Net converges:
Proposition A.5** (Sequence convergence).**
In Algorithm 2, let be convex and subdifferentiable, and be convex. Assume that the paired operators are asymptotically contractive, i.e.,
[TABLE]
with and , . Then, the sequence generated by Algorithm 2 is convergent.
Proof.
We rewrite the updates in Algorithm 2 as follows:
[TABLE]
We first show that the paired operators is asymptotically contractive:
[TABLE]
, where the inequality (A.29) holds because the subdifferential of the convex extended-valued function (the characteristic function of a convex set , , is convex, and the sum of the two convex functions, , is convex) is a monotone operator [Ryu&Boyd:16ACM:supp, §4.2], and the resolvent of a monotone relation with a positive parameter, i.e., with , is nonexpansive [Ryu&Boyd:16ACM:supp, §6], and the inequality (A.30) holds by via the contractiveness of the paired operators , . Note that the inequality (A.29) does not hold for nonconvex and/or . Considering that , we show that the sequence is Cauchy sequence:
[TABLE]
where the second inequality uses the result in (A.30). Since the sequence is Cauchy sequence, is convergent, and this completes the proofs. ∎
In terms of guaranteeing convergence, BCD-Net has three theoretical or practical limitations compared to Momentum-Net:
- –
Different from Momentum-Net, BCD-Net assumes the asymptotic contractive condition for the paired operators . When image mapping operators in (Alg.2.1) are identical across iterations, i.e., , then is assumed to be contractive. On the other hand, a mapping operator (identical across iterations) of Momentum-Net only needs to be nonexpansive. Note, however, that when with (e.g., Example 2.3), BCD-Net can guarantee the sequence convergence with the asymptotically nonexpansive paired operators (see Definition 7) [Chun&etal:19MICCAI:supp].
- –
When one applies an iterative solver to (Alg.2.2), there always exist some numerical errors and these obstruct the sequence convergence guarantee in Proposition A.5. To guarantee sufficiently small numerical errors from iterative methods solving (Alg.2.2) (so that one can find a critical point solution for the MBIR problem (Alg.2.2)), one needs to use sufficiently many inner iterations that can substantially slow down entire MBIR.
- –
BCD-Net does not guarantee the sequence convergence for nonconvex data-fit , whereas Momentum-Net guarantees convergence to a fixed-point for both convex and nonconvex .
A.8 For the sCNN architecture (18), connection between convolutional training loss (P2) and its patch-based training loss
This section shows that given the sCNN architecture (18), the convolutional training loss in (P2) has three advantages over the patch-based training loss in [Chun&Fessler:18IVMSP:supp, Chun&etal:19MICCAI:supp] that may use all the extracted overlapping patches of size :
- –
The corresponding patch-based loss does not model the patch aggregation process that is inherently modeled in (18).
- –
It is an upper bound of the convolutional loss (P2).
- –
It requires about times more memory than (P2).
We prove the benefits of (P2) using the following lemma.
Lemma A.6**.**
The loss function (P2) for training the residual convolutional autoencoder in (18) is bounded by the patch-based loss function:
[TABLE]
where the residual is defined by , are given as in (P2), and are the training data matrices whose columns are vectorized patches extracted from the images and (with the circulant boundary condition and the “stride” parameter ), respectively, is a decoding filter matrix, and is an encoding filter matrix. Here, the definition of soft-thresholding operator in (6) is generalized by
[TABLE]
for , where . See other related notations in (18).
Proof.
First, we have the following reformulation [Chun&Fessler:20TIP:supp, §S.1]:
[TABLE]
where is a permutation matrix, is defined in Lemma (A.6), and is the patch extraction operator for . Considering that via the definition of in (A.33) (see also the reformulation technique in [Chun&Fessler:20TIP:supp, §S.1]), we obtain the following reformulation result:
[TABLE]
where the soft-thresholding operators and are defined in (A.32) and we use the permutation invariance of the thresholding operator , i.e., for any . Finally, we obtain the result in (A.31) as follows:
[TABLE]
where is defined in Lemma A.6, is a set of extracted patches, the training matrices are defined by and . Here, the equality (A.35) uses the result in (A.34), the equality (A.36) holds by (for the circulant boundary condition in Lemma A.6), and the inequality (A.37) holds by with . ∎
Lemma A.6 reveals that when the patch-based training approach extract all the -size overlapping patches, 1) the corresponding patch-based loss is an upper bound of the convolutional loss (P2); 2) it requires about -times larger memory than (P2) because for and the boundary condition described in Lemma A.6, ; and 3) it misses modeling the patch aggregation process that is inherently modeled in (18) – see that the patch aggregation operator is removed in the inequality (A.37) in the proof of Lemma A.6. In addition, different from the patch-based training approach [Chun&Fessler:18IVMSP:supp, Chun&etal:19MICCAI:supp], i.e., training with the function on the right-hand side in (A.31), one can use different sizes of filters in the convolutional training loss, i.e., the function on the left-hand side in (A.31).
A.9 Details of experimental setup
A.9.1 Majorization matrix designs for quadratic data-fit
For (real-valued) quadratic data-fit in the form of , if a majorization matrix exists such that , it is straightforward to verify that the gradient of quadratic data-fit satisfies the -Lipchitz continuity in Definition 1, i.e.,
[TABLE]
because the assumption implies that the eigenspectrum of lies in the interval , and gives the following result:
[TABLE]
Next, we review a useful lemma in designing majorization matrices for a wide class of quadratic data-fit :
Lemma A.7** ([Chun&Fessler:18TIP:supp, Lem. S.3]).**
For a (possibly complex-valued) matrix and a diagonal matrix with non-negative entries, , where denotes the matrix consisting of the absolute values of the elements of .
A.9.2 Parameters for MBIR optimization models: Sparse-view CT reconstruction
For MBIR model using EP regularization, we combined a EP regularizer and the data-fit in §4.1.1, where is the set of indices of the neighborhood, and are parameters that encourage uniform noise [Cho&Fessler:15TMI:supp], and is the Lange penalty function, i.e., , with in HU. We chose the regularization parameter (e.g., in (P0)) as . We ran the relaxed linearized augmented Lagrangian method [Nien&Fessler:16:TMI:supp] with iterations and ordered-subsets, and initialized the EP MBIR algorithms with a conventional FBP method using a Hanning window.
For MBIR model using a learned convolutional regularizer [Chun&Fessler:18Asilomar:supp, (P2)], we trained convolutional regularizer with filters of via CAOL [Chun&Fessler:20TIP:supp] in an unsupervised training manner; see training details in [Chun&Fessler:20TIP:supp]. The regularization parameters (e.g., in (2.1)) were selected by applying the “spectral spread” based selection scheme in §3.2 with the tuned factor . We selected the spatial-strength-controlling hard-thresholding parameter (i.e., in [Chun&Fessler:18Asilomar:supp, (P2)]) as follows: for Test samples –, we chose it is as and , respectively. We initialized the MBIR model using a learned regularizer with the EP MBIR results obtained above. We terminated the iterations if the relative error stopping criterion (e.g., [Chun&Fessler:18TIP:supp, (44)]) is met before reaching the maximum number of iterations. We set the tolerance value as and the maximum number of iterations to .
A.9.3 Parameters for MBIR optimization models: LF photography using a focal stack
For MBIR model using 4D EP regularization [Lien&etal:20NP:supp], we combined a 4D EP regularizer and the data-fit in §4.1.2, where is the set of indices of the 4D neighborhood, and is the hyperbola penalty function, i.e., . We selected the hyperbola function parameter and regularization parameter (e.g., in (P0)) as follows: for Test samples –, we chose them as , , and , respectively. We ran the conjugate gradient method with iterations, and initialized the 4D EP MBIR algorithms with rescaled in the interval .
A.9.4 Reconstruction accuracy and depth estimation accuracy of different MBIR methods
Tables A.1–A.3 below provide reconstruction accuracy numerics of different MBIR methods in sparse-view CT reconstruction and LF photography using a focal stack, and reports the SPO depth estimation [Zhang&etal:16CVIU:supp] accuracy numerics on reconstructed LFs from different MBIR methods.
A.9.5 Reconstructed images and estimated depths with noniterative analytical methods
This section provides reconstructed images by an analytical back-projection method in sparse-view CT reconstruction and LF photography using a focal stack (see the first two columns in Fig. A.2), and estimated depths from reconstructed LFs via the SPO depth estimation method [Zhang&etal:16CVIU:supp] (see the third column in Fig. A.2(c)). Results in Fig. A.2 below are supplementary to Fig. 8, Fig. 9, and Fig. 10, and the first two columns visualize initial input images to INN methods.
A.10 How to choose parameters of image refining modules in soft-refining INNs?
In soft-refining INNs using iterative-wise refining NNs, one does not need to greatly increase parameter dimensions of refining NNs [Lim&etal:20TMI:supp, Chun&etal:19MICCAI:supp]. The natural question then arises, “How one can choose between sCNN (18) and dCNN (19) refiners, and select their parameters (, , and )?” The first answer to this question depends on some understanding of data-fit in MBIR problem (P1), e.g., the regularization strength and the condition number variations across training data-fit majorizers. (An additional criteria could be general understandings between sample size/diversity and parameter dimension of NNs.)
For example, the sparse-view CT system in §4.1.1 needs moderate regularization strength () and the majorization matrices of its training data-fits have mild condition number variations (the standard deviation is ). training data-fits have mild parameter variations across samples. Comparing results between Momentum-Net-sCNN and -dCNN in Fig. 5 and Table A.1 demonstrates that sCNN (18) seems suffice. Table A.1(d)–(e) shows that one can further improve the refining accuracy of sCNN (18) by increasing its width, i.e., . The LF photography system using a limited focal stack in §4.1.2 needs a large value (), and the majorization matrices of its training data-fits have large condition number variations (the standard deviation is ). Comparing results between Momentum-Net-sCNN and -dCNN in Fig. 7 and Table A.2 demonstrates that dCNN (19) yields higher PSNR than sCNN (18). For dCNN (19), we observed increasing its depth, i.e., , up to a certain number is more effective than increasing its width, i.e., , as briefly discussed in §4.2.1.
For choosing the relaxation parameter in (Alg.1.1), we also suggest considering the regularization strength in (5). For an application that needs moderate regularization strength, e.g., sparse-view CT in §4.1.1, we suggest setting to so as to mix information between input and output of refining NNs, rather than that does not mix input and output. For an application that needs strong regularization, e.g., LF photography using a limited focal stack in §4.1.2, we suggest using than . Results in the next section validate this suggestion.
A.10.1 Performance of Momentum-Net with different relaxation parameters in (Alg.1.1)
Fig. A.3 below compares the performances of Momentum-Net-sCNN with different values. The results in Fig. A.3 support the selection guideline in §4.2.3. One can maximize the MBIR accuracy of Momentum-Net by properly selecting .
Note that controls strength of inference from refining NNs in (Alg.1.1), but does not affect the convergence guarantee of Momentum-Net. Fig. A.3 illustrates that Momentum-Net appears to converge regardless of values.
A.11 Parameters of Momentum-Net
Table A.4 below lists parameters of Momentum-Net, and summarizes selection guidelines or default values. Similar to BCD-Net/ADMM-Net, the main tuning jobs to maximize the performance of Momentum-Net include selecting architectures of refining NNs in (Alg.1.1), and choosing a regularization parameter in (5) by tuning in §3.2. One can simplify the tuning process by using the selection guidelines in §A.10 for selecting architectures of , and training in §3.2. Note that one designs majorization matrices rather than tuning them: majorization matrices can be analytically designed, e.g., Lemma A.7 as used in §4.2.1; one can algorithmically design them [Mcgaffin&Fessler:15arXiv:supp]. Tighter majorization matrices are expected to further accelerate the convergence of Momentum-Net [Chun&Fessler:18TIP:supp, Chun&Fessler:20TIP:supp].
\bibliographystyleSupp
IEEEtran \bibliographySuppreferencesSupp_Bobby
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol, “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion,” J. Mach. Learn. Res. , vol. 11, no. Dec, pp. 3371–3408, 2010.
- 2[2] J. Xie, L. Xu, and E. Chen, “Image denoising and inpainting with deep neural networks,” in Proc. NIPS , Lake Tahoe, NV, Dec. 2012, pp. 341–349.
- 3[3] X. Mao, C. Shen, and Y.-B. Yang, “Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections,” in Proc. NIPS , Barcelona, Spain, Dec. 2016, pp. 2802–2810.
- 4[4] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising,” IEEE Trans. Image Process. , vol. 26, no. 7, pp. 3142–3155, Feb. 2017.
- 5[5] L. Xu, J. S. Ren, C. Liu, and J. Jia, “Deep convolutional neural network for image deconvolution,” in Proc. NIPS , Montreal, Canada, Dec. 2014, pp. 1790–1798.
- 6[6] J. Sun, W. Cao, Z. Xu, and J. Ponce, “Learning a convolutional neural network for non-uniform motion blur removal,” in Proc. IEEE CVPR , Boston, MA, Jun. 2015, pp. 769–777.
- 7[7] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 38, no. 2, pp. 295–307, Feb. 2016.
- 8[8] J. Kim, K. J. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in Proc. IEEE CVPR , Las Vegas, NV, Jun. 2016.