Joint Reconstruction via Coupled Bregman Iterations with Applications to PET-MR Imaging

TL;DR

This paper introduces a novel joint reconstruction method for multi-modality medical imaging, specifically PET-MRI, using generalized Bregman distances and infimal convolutions to effectively handle images with different scales and improve reconstruction quality.

Contribution

The proposed method employs total variation subgradients and a weighting scheme to enhance multi-modality image reconstruction, addressing scale differences and outperforming existing techniques.

Findings

Superior reconstruction quality compared to separate methods

Effective handling of scale differences in multi-modality images

Mutual benefit observed in PET and MRI reconstructions

Abstract

Joint reconstruction has recently attracted a lot of attention, especially in the field of medical multi-modality imaging such as PET-MRI. Most of the developed methods rely on the comparison of image gradients, or more precisely their location, direction and magnitude, to make use of structural similarities between the images. A challenge and still an open issue for most of the methods is to handle images in entirely different scales, i.e. different magnitudes of gradients that cannot be dealt with by a global scaling of the data. We propose the use of generalized Bregman distances and infimal convolutions thereof with regard to the well-known total variation functional. The use of a total variation subgradient respectively the involved vector field rather than an image gradient naturally excludes the magnitudes of gradients, which in particular solves the scaling behavior. Additionally, the presented method features a weighting that allows to control the amount of interaction between channels. We give insights into the general behavior of the method, before we further tailor it to a particular application, namely PET-MRI joint reconstruction. To do so, we compute joint reconstruction results from blurry Poisson data for PET and undersampled Fourier data from MRI and show that we can gain a mutual benefit for both modalities. In particular, the results are superior to the respective separate reconstructions and other joint reconstruction methods.

Tables

Table 1. Table 1: SSIM values for single and joint reconstruction results for different samplings. The parameters λ 𝜆 \lambda and μ 𝜇 \mu are the weighting of the own channel during the joint reconstruction with ICB. E.g. for ’half’, the influence of MRI during the PET reconstruction was 70%, the influence of PET on the MRI reconstruction was 20%.

	Full		Half		Spokes		Spiral
	PET	MRI	PET	MRI	PET	MRI	PET	MRI
no prior	$0.5366$	$0.9940$	$0.5366$	$0.6722$	$0.5366$	$0.7804$	$0.5366$	$0.5283$
TV	$0.5657$	$0.9942$	$0.5657$	$0.6718$	$0.5657$	$0.9415$	$0.5657$	$0.7807$
JTV	$0.6806$	$0.9944$	$0.6538$	$0.8194$	$0.6340$	$0.8977$	$0.6540$	$0.9101$
LPLS	$0.7837$	$0.9949$	$0.6812$	$0.8322$	$0.6485$	$0.8541$	$0.6485$	$0.8537$
ICB	$0.9218$	$0.9999$	$0.8025$	$0.8969$	$0.7695$	$0.9453$	$0.7017$	$0.8240$
$\lambda$ / $\mu$	$0.1$	$1$	$0.3$	$0.8$	$0.1$	$1$	$0.2$	$0.6$

Equations

$$Ku=f,$$

$$K_i{u}_i=f_i.$$

$${\int }_{\Omega }|\nabla u||\nabla v|-|\nabla u\cdot \nabla v|\mathrm{d}x={\int }_{\Omega }|\nabla u||\nabla v|(1-|\cos (\phi )|)\mathrm{d}x,$$

$$\displaystyle\textnormal{TV}(\boldsymbol{u})=\Big{\|}\|\nabla\boldsymbol{u}\|_{M}\Big{\|}_{1}=\int_{\Omega}\|\nabla\boldsymbol{u}(x)\|_{M}\,\mathrm{d}x.$$

$${\int }_{\Omega }\Vert \nabla \mathbf{u}(x){\Vert }_2\mathrm{d}x={\int }_{\Omega }{\left({\sum }_i|\nabla u_i(x){|}^2\right)}^{\frac{1}{2}}\mathrm{d}x,$$

$${\text{D}}_J^p(v,u)=J(v)-J(u)-⟨p,v-u⟩,p\in \partial J(u).$$

$$D_{\text{TV}}^p(v,u)={\int }_{\Omega }|\nabla v|\left(1-\frac{\nabla v}{|\nabla v|}\cdot \frac{\nabla u}{|\nabla u|}\right)\mathrm{d}x={\int }_{\Omega }|\nabla v|\left(1-\cos (\phi )\right)\mathrm{d}x,$$

$$\displaystyle u_{i}^{k+1}\in\operatorname*{arg\,min}_{u_{i}\in\textnormal{BV}(\Omega)}\Bigg{\{}\alpha_{i}H_{f_{i}}(K_{i}u_{i})+\sum_{j=1}^{N}w_{ij}D_{\textnormal{TV}}^{p_{j}^{k}}(u_{i},u_{j}^{k})\Bigg{\}},$$

$$\displaystyle\mathrm{ICB}_{\mathrm{TV}}^{p}(v,u):=[\textnormal{D}_{\textnormal{TV}}^{p}(\cdot,u)\Box\textnormal{D}_{\textnormal{TV}}^{-p}(\cdot,-u)](v)=\inf_{v=\phi+\psi}\Big{\{}\textnormal{D}_{\textnormal{TV}}^{p}(\phi,u)+\textnormal{D}_{\textnormal{TV}}^{-p}(\psi,-u)\Big{\}}.$$

$$u_i^{k+1}\in \underset{u_i\in \text{BV}(\Omega )}{\mathrm{argmin}}$$

$${\ell }^1({\mathbb{R}}^m)$$

$${\ell }^{\infty }({\mathbb{R}}^m)$$

$$⟨q,\eta ⟩:=\sum _{i=1}^{\infty }{q}_i\cdot {\eta }_i.$$

$$0=\cos ({\phi }_i)|{\nu }_i||{\eta }_i|-{\nu }_i\cdot {\eta }_i\le |{\nu }_i||{\eta }_i|-{\nu }_i\cdot {\eta }_i,$$

$$\sum _{i=1}^{\infty }|{\nu }_i||{\eta }_i|-{\nu }_i\cdot {\eta }_i.$$

$$\partial \Vert \nu {\Vert }_{{\ell }^1({\mathbb{R}}^m)}=\left\{q\in {\ell }^{\infty }({\mathbb{R}}^m):|q_i|\le 1,q_i=\frac{{\nu }_i}{|{\nu }_i|}\text{ if }{\nu }_i\ne 0\right\}.$$

$$D_{\Vert \cdot {\Vert }_{{\ell }^1({\mathbb{R}}^m)}}^q(\eta ,\nu )=\Vert \eta {\Vert }_{{\ell }^1({\mathbb{R}}^m)}-⟨q,\eta ⟩=\sum _{i=1}^{\infty }|{\eta }_i|-q_i\cdot {\eta }_i=\sum _{i=1}^{\infty }|{\eta }_i|(1-\cos ({\phi }_i)|q_i|).$$

$$|\nu |(E):=\sup \left\{\sum _{i=1}^{\infty }|\nu (E_i)|:(E_i{)}_{i\in \mathbb{N}}\subset \mathcal{B}(\Omega )\text{ pairwise disjoint, }E=\bigcup _{i=1}^{\infty }{E}_i\right\}.$$

$$⟨\nu ,\varphi ⟩={\int }_{\Omega }\varphi \cdot \mathrm{d}\nu :=\sum _{i=1}^m{\int }_{\Omega }{\varphi }_i\mathrm{d}{\nu }_i,$$

$$\nu (E)={\int }_E{f}_{\nu }\mathrm{d}|\nu |.$$

$$⟨q,\eta -\nu ⟩\le \Vert \eta {\Vert }_{\mathcal{M}}-\Vert \nu {\Vert }_{\mathcal{M}}$$

$${\delta }_x(E)=\{\begin{array}{ll}1,& x\in E\\ 0,& \text{else}\end{array}$$

$$⟨q,{\nu }_{\epsilon }-\nu ⟩\le \Vert {\nu }_{\epsilon }{\Vert }_{\mathcal{M}}-\Vert \nu {\Vert }_{\mathcal{M}}\Rightarrow ⟨q,\sigma {\delta }_x⟩\le 1.$$

$$\underset{\sigma \in S^{m-1}}{\sup }|⟨q,\sigma {\delta }_x⟩|=\underset{\sigma \in S^{m-1}}{\sup }\left|{\int }_{\Omega }q\cdot \sigma \mathrm{d}{\delta }_x\right|=\underset{\sigma \in S^{m-1}}{\sup }|(q\cdot \sigma )(x)|\le 1$$

$$⟨q,\nu ⟩=\Vert \nu {\Vert }_{\mathcal{M}}\iff {\int }_{\Omega }q\cdot \mathrm{d}\nu ={\int }_{\Omega }1\mathrm{d}|\nu |\iff {\int }_{\Omega }q\cdot f_{\nu }\mathrm{d}|\nu |={\int }_{\Omega }1\mathrm{d}|\nu |,$$

$$⟨q,\eta -\nu ⟩$$

$$\le \Vert q{\Vert }_{\infty }{\int }_{\Omega }1\mathrm{d}|\eta |-{\int }_{\Omega }1\mathrm{d}|\nu |$$

$$D_{\Vert \cdot {\Vert }_{\mathcal{M}}}^q(\eta ,\nu )=\Vert \eta {\Vert }_{\mathcal{M}}-⟨q,\eta ⟩={\int }_{\Omega }1\mathrm{d}|\eta |-{\int }_{\Omega }q\cdot f_{\eta }\mathrm{d}|\eta |={\int }_{\Omega }1-q\cdot f_{\eta }\mathrm{d}|\eta |.$$

$$D_{\Vert \cdot {\Vert }_{\mathcal{M}}}^q(\eta ,\nu )={\int }_{\Omega }1-\cos (\phi )|q|\mathrm{d}|\eta |,$$

$${\int }_{\Omega }u\text{div}(\varphi )\mathrm{d}x=-\sum _{i=1}^m{\int }_{\Omega }{\varphi }_i\mathrm{d}{\mathrm{D}}_{i}u\text{ for all }\varphi \in C_c^{\infty }(\Omega ,{\mathbb{R}}^m).$$

$$\text{TV}(u):=\underset{\begin{array}{c}\varphi \in C_c^{\infty }(\Omega ,{\mathbb{R}}^m),\\ \Vert \varphi {\Vert }_{\infty }\le 1\end{array}}{\sup }{\int }_{\Omega }u\text{div}(\varphi )\mathrm{d}x<\infty .$$

Full text

**Keywords: ** Joint reconstruction, Bregman iterations, positron emission tomography, magnetic resonance imaging, structural similarity, infimal convolution of Bregman distances, total variation

1 Introduction

In the last century, the need for better medical diagnostics has been the driving force for developing a large variety of medical imaging systems. Accordingly, nowadays it is common practice to perform several experiments on the same patient, e.g. MRI, PET and CT scans, in order to reduce the risk of false interpretation of the measured data. To do so, however, one needs reconstruction methods that can exploit the connections between different types of data sets efficiently.

In general, a wide range of imaging applications requires the solution of an inverse problem

[TABLE]

where typically $u\colon\Omega\subset\mathbb{R}^{d}\to\mathbb{R}$ describes a property such as a density of the investigated unknown object. A common issue for most of these problems is a general ill-posedness, which makes it difficult to solve for $u$. This is mainly due to the properties of the imaging operator $K$ and the possibly poor quality of the available data $f$. In order to nevertheless obtain a reasonable solution, it is helpful to gather additional knowledge about $u$. This knowledge can be a rather abstract assumption on its properties as a mathematical object, such as being smooth, piecewise constant, sparse etc., which is often encoded in the choice of function spaces for the inversion of (1.1). This process is usually referred to as regularization. On the contrary one could as well imagine to extend the knowledge about the investigated object by performing several experiments, such that one has access to multiple data sets $f_{i}$, and one has to solve a series of inverse problems for $i=1,\dots,N$,

[TABLE]

This includes the case of a simple repetition of the same experiment as (1.1), but as well a change of the experimental setup or the imaging modality, such that one has very different imaging operators $K_{i}$ and resulting data sets $f_{i}$.

Whenever one can assume a relation between the $u_{i}$, one can try to exploit that connection in order to improve the individual inversions. The process and the challenge of coupling several inverse problems during the inversion is known as joint reconstruction or joint inversion [21, 24, 25, 26, 27, 28, 29], and has attracted a lot of attention in geophysical applications, medical imaging and color image processing.

The approaches can essentially be divided into two classes: either some of the $u_{i}$ are known a-priori and serve as additional information for the reconstruction of the others in a static mode, or we aim to recover all $u_{i}$ simultaneously. The former is sometimes referred to as model fusion [28] or structural and anatomical priors [5, 14, 31, 41]. It arises e.g in atlas-based reconstructions or attenuation correction in PET-CT, where the attenuation of photons is estimated via a CT image and included in the reconstruction of the PET image [52]. Another application is anatomical MRI priors for quantitative PET imaging [2, 35, 37, 50]. We will herein focus on a simultaneous reconstruction of all $u_{i}$, though small adaptions of the proposed approach as well allow for the use of static priors to perform model fusion.

A wide range of recently developed techniques makes use of a similar structure or shape of the images. Assuming a shared edge set, a common approach is to compare the gradients of the images with regard to position, direction and magnitude. For example, Ehrhardt et al. [21, 19] derived a measure for the similarity of image gradients, called Parallel Level Sets (PLS), using the definition of the Euclidean inner product:

[TABLE]

where $\varphi$ denotes the angle between $\nabla u$ and $\nabla v$. A variant thereof is related to a work of Gallardo et al. [25], using the cross-product of gradients as a measure for structural similarity of images. Yet another related approach on coupling of image structures has been proposed by Knoll et al. [33] who extended the well-known TV functional [45] and its extensions [6] to vector-valued data $\boldsymbol{u}=(u_{1},\dots,u_{n})$, e.g.

[TABLE]

The important part is then the choice of an appropriate matrix norm $\|\cdot\|_{M}$ for the point-wise coupling of the gradients. A Euclidean norm (or Frobenius norm) leads to a joint TV setting, i.e.

[TABLE]

which promotes joint positions of non-zero elements of the gradients. Another possible choice is the nuclear norm, i.e. the sum of the singular values of $\nabla\boldsymbol{u}$, which prefers linearly dependent image gradients.

All these methods perform well on the positioning and alignment of image gradients. However, since they all depend on the magnitude of the gradients, they have difficulties comparing images on different scales, in particular if a global scaling of the data or images is not possible. This is e.g. the case if the images share some jumps of equal height, but one image also features small jumps where the others have a large one. During the reconstruction, this leads to a penalization of the gradient height (and hence image intensities) rather than to an alignment of image gradients. This is a significant drawback and still an open issue.

A particular application where this problem arises is the coupling of Positron Emission Tomography (PET) and Magnetic Resonance Imaging (MRI), since by now both techniques are available in a single device. They are able to simultaneously gather functional PET data and structural MR data, which are adjusted in spatial and temporal terms. Though inherently different in their physical interpretation, both data sets stem from the same investigated object. As function is not independent from the underlying anatomy [21], we can hence assume a similar structure in both the PET and MR image. However, the images tend to have entirely different scales, and jumps across the edges do not share the same height and sign.

To overcome that issue, we reformulate a recent work on color image processing [39] and further extend it to a general joint reconstruction approach using the notion of generalized Bregman distances [10]:

[TABLE]

In case of $J=\textnormal{TV}$ the Bregman distance can formally be rewritten as

[TABLE]

with $\varphi$ denoting the angle between $\nabla u$ and $\nabla v$. Hence, from a geometric viewpoint, it penalizes the total variation of the first argument $v$, weighted by the alignment of its gradient to the gradient of the second argument $u$. In particular, there is no penalty for entirely aligned image gradients ($\cos(\varphi)=1$), independent of the height of the jump in $v$. Equally relevant is the normalization of the gradient of $u$, which excludes its magnitude from the functional. This is the key feature that allows for the comparison of images in different scales. We point out that the Bregman distance obviously is an asymmetric measure which uses the subgradient as a-priori information. We hence follow [39] and propose the following iterative scheme for a joint reconstruction:

[TABLE]

$i=1,\dots,N$. Here, the data fidelity terms $H_{f_{i}}$ enforce a closeness of $u_{i}$ to the data $f_{i}$ and the parameters $\alpha_{i}$ balance between data accuracy and regularization. In each step of the procedure we solve the latter problem for one $u_{i}^{k+1}$, using the subgradients of the other $u_{j}^{k}$ from the previous iteration as a-priori edge information, thus fitting the edge sets of $u_{i}^{k+1}$ to the edge sets of all $u_{j}^{k}$. The matrix $W=(w_{ij})\in\mathbb{R}^{N\times N}$ serves as a weighting between the different channels and is chosen such that its rows are normalized. It allows for different amounts of influence of the different channels and can be adapted to the particular joint reconstruction situation, i.e. according to different data accuracy in different channels, number of channels etc.

However, the iteration scheme (1.5) features a drawback. A closer look at equation (1.4) reveals that it highly penalizes opposing vectors ($\cos(\varphi)=-1$). Simply speaking, it favors that jumps across the edges share the same sign. In view of different image contrasts, e.g. for medical multi-modality imaging, this is too restrictive. We therefore again follow [39] and employ the use of the infimal convolution of Bregman distances, to gain a measure favoring linearly dependent image gradients without the edge orientation constraint:

[TABLE]

That way we can find a local decomposition of $v$ such that one part matches the subgradient $p$, and the other part matches the negative subgradient $-p$. Without further explanation at this point, we propose the following iteration scheme for a joint reconstruction:

[TABLE]

Similarly to (1.5) we enforce a common edge set of the $u_{i}$ during the iterations, now excluding the orientation of edges. We point out the use of the Bregman distance on the own channel (or the diagonal), since the edge orientations should matter here.

The contributions of this paper are twofold: on the theoretical side we analyze the structure and behavior of Bregman distances and line out their applicability to a structural joint reconstruction. We start with the case of infinite $\ell^{1}$-sequences and the $\ell^{1}$-norm, and generalize the results to a continuous setting using the notion of finite Radon measures. In particular we compute a closed-form representation of the infimal convolution of two Bregman distances with respect to the total variation of a Radon measure and subgradients with opposing sign. These results are used to finally investigate Bregman distances of distributional derivatives of functions of bounded variation. On the applied side we show how to solve the derived method numerically, and apply it to the recent problem of PET-MRI joint reconstruction. We line out the necessary mathematical modeling of the joint reconstruction problem, reformulate our method in this context and show numerical results on artificial PET-MR data. In particular we compare with two recent existing joint reconstruction methods, namely Parallel Level Sets [21] and Joint Total Variation [28].

The remainder of the paper is organized as follows: In Section 2 we derive the method in detail from a geometric viewpoint and describe its behavior analytically, before we show how to numerically solve it in Section 3. In Section 4 we then outline the idea and modeling of PET-MRI joint reconstruction and present our method tailored to that particular application. We continue with numerical results for PET-MRI joint reconstruction and then give a short conclusion and outlook in Section 5.

2 Joint reconstruction via Bregman distances

In this chapter we investigate the use of Bregman distances and infimal convolutions thereof as methods for structural joint reconstruction. We start in a discrete setting with the definition of angles between vectors in a Euclidean space and show their connection to a Bregman distance with respect to the $\ell^{1}$-norm. Subsequently, we generalize the idea to the continuous setting using the notion of finite Radon measures, in order to finally compare the (distributional) gradients of functions of bounded variation. Finally, we further motivate the use of the infimal convolution in order to eliminate the occurring edge orientation constraint.

2.1 The discrete case: vector sequences

We aim for a regularizer being able to link the edges or level sets of several real-valued functions, but restrict ourselves to two in the following. Simply speaking, the gradient of a function is perpendicular to its level sets. Hence we want to compare the geometric features of gradients as a vector, i.e. their location, direction and magnitude, and align them in order to obtain a similar structure in the different image channels. For the discrete setting let us introduce the space of summable respectively bounded $\mathbb{R}^{m}$-valued sequences:

Definition 1**.**

For the Euclidean norm $|\cdot|$ on $\mathbb{R}^{m}$ and $m\in\mathbb{N}$ we define

[TABLE]

Note that $\ell^{\infty}(\mathbb{R}^{m})$ is the dual of $\ell^{1}(\mathbb{R}^{m})$ under the pairing

[TABLE]

So let $\nu,\eta\in\ell^{1}(\mathbb{R}^{m})$ be two summable sequences of vectors, and let $\varphi_{i}$ denote the angle between $\nu_{i}$ and $\eta_{i}$ ($\varphi_{i}:=0$ if one of the vectors is zero). Then we know by the geometric definition of the Euclidean inner product that the following relation holds true:

[TABLE]

with equality if and only if $\nu_{i}$ and $\eta_{i}$ are parallel ($\cos(\varphi_{i})=1$), and maximum value on the right hand side if and only if $\nu_{i}$ and $\eta_{i}$ are anti-parallel ($\cos(\varphi_{i})=-1$). Hence we can consider the right-hand side of (2.1) as a symmetric measure for the alignment of vectors which increases with the deviation of the vectors. We can easily extend it for all $i$ by summing up:

[TABLE]

However, its use for measuring parallelism of vectors imposes some issues. Since it measures the magnitude of both vectors as well, we are facing scaling issues for vectors of different magnitudes (or image gradients for images on different scales). Additionally, the measure vanishes if one of the vectors is zero, which results in no coupling if one of the images is flat. Note that technically this is an advantage for joint reconstruction, since it avoids the artificial transfer of non-shared structures between the images. However, in presence of degraded and noisy data the vanishing of either of the vectors would result in no regularization for the other one, which is not desirable.

We shall instead have a look at specific subgradients rather than gradients to overcome the above issues. Recall that the subdifferential of the (isotropic) $\|\cdot\|_{\ell^{1}(\mathbb{R}^{m})}$-norm at $\nu$ is given by

[TABLE]

The associated Bregman distance between $\eta$ and $\nu$ and $q\in\partial\|\nu\|_{\ell^{1}(\mathbb{R}^{m})}$ reads

[TABLE]

We can distinguish two situations. If $\nu_{i}\neq 0$, then $q_{i}=\nu_{i}/|\nu_{i}|$, hence the Bregman distance penalizes the magnitude of $\eta_{i}$, weighted by its deviation from $\nu_{i}$ in terms of the included angle $\varphi_{i}$. In particular, there is no penalty on $|\eta_{i}|$ if $\eta_{i}$ and $\nu_{i}$ are aligned. We as well remark that the magnitude of $\nu_{i}$ is excluded from the functional, which allows to compare vectors of different magnitude. If $\nu_{i}=0$, the associated subgradient is an element of the unit ball. In order to minimize, we can again distinguish two cases: For a rather small magnitude of $q_{i}$, the benefit of aligning $q_{i}$ and $\eta_{i}$ is rather small in comparison to a small $|\eta_{i}|$, so we are more likely to simply penalize $|\eta_{i}|$. The larger $q_{i}$ gets, the more beneficial it is to actually align $q_{i}$ and $\eta_{i}$.

At first sight, this concept does not seem to be intuitive, since in theory $q$ can be chosen arbitrarily from the unit ball, and there is no reason to align $\eta_{i}$ to an arbitrary direction. However, in practice we do not choose a random $q$ but instead the procedure itself yields a specific subgradient. For standard Bregman iterations [10, 42], the subgradient is chosen in relation to the data residual (cf. Equation (2.8)), thus containing more than only information about the last iterate. Accordingly, as already indicated in [39], the subgradient as well serves as an indicator for structures which are likely to appear in the next iterate. Hence, thinking of the vectors as image gradients, the length of $q$ can be interpreted as the possibility of an edge at that location, while its direction implies the direction of the jump across that edge.

2.2 The continuous case: $\mathbb{R}^{m}$-valued Radon measures

In this section we transfer the idea introduced above to a continuous setting. Since for a joint reconstruction we want to compare the gradients of functions, natural solution spaces are the Sobolev spaces of weakly differentiable functions, e.g. $W^{1,1}(\Omega)$. However, these functions do not permit discontinuities. In order to allow for jump discontinuities, and hence sharp edges in the reconstructions, the canonical choice is the space of functions of bounded variation. Since the (now distributional) derivative of such a function defines a finite Radon measure, the natural extension of the above concept turns out to be the space of finite Radon measures, for which we shall find an analogous result to the last section. Let $d,m\in\mathbb{N}$ and let $\Omega\subset\mathbb{R}^{d}$ be a bounded and open set.

Definition 2**.**

Let $\mathcal{B}(\Omega)$ be the Borel $\sigma$-algebra generated by the open sets in $\Omega$. A mapping $\nu\colon\mathcal{B}(\Omega)\to\mathbb{R}^{m}$ is called an $\mathbb{R}^{m}$-valued, finite Radon measure on $\Omega$, if it is $\sigma$-additive and $\nu(\emptyset)=0$. For every set $E\in\mathcal{B}(\Omega)$ we define the total variation $|\nu|\colon\mathcal{B}(\Omega)\to[0,\infty)$ of $\nu$ as

[TABLE]

Furthermore we denote the space of all finite $\mathbb{R}^{m}$-valued Radon measures on $\Omega$ by $\mathcal{M}(\Omega,\mathbb{R}^{m})$.

We shall need the characterization of $\mathcal{M}(\Omega,\mathbb{R}^{m})$ as a dual space, which can be found in [1, Thm. 1.54].

Proposition 1**.**

The space $\mathcal{M}(\Omega,\mathbb{R}^{m})$ equipped with $\|\nu\|_{\mathcal{M}}:=|\nu|(\Omega)$ is a Banach space. It can be considered as the dual of $C_{0}(\Omega,\mathbb{R}^{m})$ under the pairing

[TABLE]

where $\nu=(\nu_{1},\dots,\nu_{m})\in\mathcal{M}(\Omega,\mathbb{R}^{m})$ and $\phi=(\phi_{1},\dots,\phi_{m})\in C_{0}(\Omega,\mathbb{R}^{m})$. Furthermore, $\|\cdot\|_{\mathcal{M}}$ is the dual norm.

As a direct consequence of the Radon-Nikodým theorem, the polar decomposition of a finite $\mathbb{R}^{m}$-valued Radon measure yields a decomposition into its direction and magnitude (cf. [1]).

Proposition 2**.**

Let $\nu\in\mathcal{M}(\Omega,\mathbb{R}^{m})$. Then there exists a function $f_{\nu}\in L_{|\nu|}^{1}(\Omega,\mathbb{R}^{m})$ such that $|f_{\nu}|=1$ $|\nu|$-a.e. and for every $E\in\mathcal{B}(\Omega)$

[TABLE]

$f_{\nu}$* is unique $|\nu|$-a.e. and we shall refer to it as the density or direction of $\nu$.*

We first compute the subdifferential $\partial\|\cdot\|_{\mathcal{M}}\subset\mathcal{M}(\Omega,\mathbb{R}^{m})^{*}$ of the total variation. Since the dual space $\mathcal{M}(\Omega,\mathbb{R}^{m})^{*}$ possesses a rather difficult structure (cf. [32]), a thorough study of the subdifferential goes beyond the scope and intention of this paper. We shall thus consider elements in $\mathcal{C}(\nu):=\partial\|\nu\|_{\mathcal{M}}\cap C_{0}(\Omega,\mathbb{R}^{m})$, i.e. subgradients that can be identified with a continuous function via the canonical embedding of the predual space $C_{0}(\Omega,\mathbb{R}^{m})$ into $\mathcal{M}(\Omega,\mathbb{R}^{m})^{*}$. It is clear that $\mathcal{C}(\nu)$ can be empty for arbitrary measures $\nu\in\mathcal{M}(\Omega,\mathbb{R}^{m})$. Hence, for the remainder of this paper we restrict ourselves to measures $\nu$ such that $\mathcal{C}(\nu)\neq\emptyset$, which immediately implies that the associated density function $f_{\nu}$ is continuous $|\nu|$-a.e. This can be seen directly from the next theorem.

Remark 1**.**

We remark that without the assumption $\mathcal{C}(\nu)\neq\emptyset$ very little can be said about the dual pairing between a measure and an element from its dual space, and hence about the structure of the associated Bregman distance. The assumption mainly serves illustrative purposes, since it allows to easily interpret the behavior of the method we derive in the course of this section. The final method however does not depend on the existence of a continuous subgradient (see also Remark 2).

Theorem 1**.**

Let $\nu\in\mathcal{M}(\Omega,\mathbb{R}^{m})$ with polar decomposition $\nu=f_{\nu}|\nu|$ such that $\mathcal{C}(\nu):=\partial\|\nu\|_{\mathcal{M}}\cap C_{0}(\Omega,\mathbb{R}^{m})\neq\emptyset$. Then $q\in\mathcal{C}(\nu)$ if and only if $q\in C_{0}(\Omega,\mathbb{R}^{m})$ with $\|q\|_{\infty}\leq 1$ and $q=f_{\nu}~{}|\nu|$-a.e.

Proof.

First, let $q\in\mathcal{C}(\nu)$. Then by the definition of the subdifferential we have

[TABLE]

for all $\eta\in\mathcal{M}(\Omega,\mathbb{R}^{m})$. In particular, for arbitrary $x\in\Omega$ let the Dirac measure $\delta_{x}\colon\mathcal{B}(\Omega)\to\mathbb{R}$ be defined by

[TABLE]

for all $E\in\mathcal{B}(\Omega)$. Then for any $\varepsilon>0$ and fixed $\sigma\in S^{m-1}=\{\sigma\in R^{m}~{}|~{}|\sigma|=1\}$, $\varepsilon\sigma\delta_{x}\in\mathcal{M}(\Omega,\mathbb{R}^{m})$ defines a finite Radon measure. Hence for $\nu_{\varepsilon}:=\nu+\varepsilon\sigma\delta_{x}$ we find $\|\nu_{\varepsilon}\|_{\mathcal{M}}\leq\|\nu\|_{\mathcal{M}}+\varepsilon$ and

[TABLE]

Since $\sigma$ was arbitrary we deduce

[TABLE]

which yields $|q(x)|\leq 1$ for the particular choice $\sigma=\frac{q}{|q|}(x)$. Since $x$ was arbitrary we deduce that $\|q\|_{\infty}\leq 1$. Furthermore using $\eta=0$ and $\eta=2\nu$ in (2.3) we obtain

[TABLE]

hence $1=q\cdot f_{\nu}$ $|\nu|$-a.e. (note that $1-q\cdot f_{\nu}\geq 0$ $|\nu|$-a.e.). Since $|f_{\nu}|=1$, this implies $q=f_{\nu}$ $|\nu|$-a.e. and in particular that $f_{\nu}$ is continuous $|\nu|$-a.e.

On the other hand, for any $q\in C_{0}(\Omega,\mathbb{R}^{m})$ with $\|q\|_{\infty}\leq 1$ and $q=f_{\nu}$ $|\nu|$-a.e. we calculate for any $\eta\in\mathcal{M}(\Omega,\mathbb{R}^{m})$:

[TABLE]

which yields the assertion. ∎

Note that the structure of the $\|\cdot\|_{\mathcal{M}}$-subdifferential is similar to the structure of the $\ell^{1}$-subdifferential from the last subsection. A subgradient $q\in\mathcal{C}(\nu)$ equals the direction $f_{\nu}$ of the measure $\nu$ on its support, while it is an arbitrary element from the unit ball on the zero sets of $|\nu|$.

Let now $\eta,\nu\in\mathcal{M}(\Omega,\mathbb{R}^{m})$ be two $\mathbb{R}^{m}$-valued Radon measures with polar decomposition $\eta=f_{\eta}|\eta|,\nu=f_{\nu}|\nu|$ such that $\mathcal{C}(\nu)\neq\emptyset$. We can immediately write down the Bregman distance between $\eta$ and $\nu$ with respect to $q\in\mathcal{C}(\nu)$ to obtain an analogous result to the discrete case.

[TABLE]

We again denote the angle between $q$ and $f_{\eta}$ by $\varphi$,

[TABLE]

and can again distinguish two situations to minimize (2.4), now in an $|\eta|$-a.e. sense. In case $q=f_{\nu}$, i.e. on the support of $\nu$, it is beneficial to either align $f_{\eta}$ to $f_{\nu}$ or to assign no mass to $\eta$. If $|q|<1$, i.e. outside the support of $\nu$, it does not provide any explicit side information. Instead, depending on the magnitude of $q$ it is either possible to assign no mass to $\eta$, or to align its direction $f_{\eta}$ to $q$.

We shall finally transfer the concept to the (distributional) gradients of $\textnormal{BV}(\Omega)$-functions in order to compare the structures of images.

Definition 3**.**

A function $u\in L^{1}(\Omega)$ is a function of bounded variation in $\Omega$ if and only if its distributional derivative is representable by a finite Radon measure $\mathrm{D}u=(\mathrm{D}_{1}u,\dots,\mathrm{D}_{m}u)$ in $\Omega$, i.e.

[TABLE]

We denote the vector space of all functions of bounded variation by $\textnormal{BV}(\Omega)$. An equivalent formulation (see [1, Prop. 3.6.]) is that $u\in\textnormal{BV}(\Omega)$ if and only if

[TABLE]

In particular, $\textnormal{TV}(u)=\|\mathrm{D}u\|_{\mathcal{M}}$.

Hence let $\mathrm{D}\colon\textnormal{BV}(\Omega)\to\mathcal{M}(\Omega,\mathbb{R}^{m})$ denote the distributional derivative. We obtain the following result.

Proposition 3**.**

Let $u,v\in\textnormal{BV}(\Omega)$ with derivatives $\mathrm{D}u,\mathrm{D}v\in\mathcal{M}(\Omega,\mathbb{R}^{m})$, and $p\in\partial\textnormal{TV}(u)$. Then

[TABLE]

for some $q\in\partial\|\mathrm{D}u\|_{\mathcal{M}}$.

Proof.

Since $\mathrm{D}$ is continuous between $\textnormal{BV}(\Omega)$ and $\mathcal{M}(\Omega,\mathbb{R}^{m})$ and the underlying functionals are convex and lower semicontinuous, the chain rule [22, p.27] yields $p\in\partial\textnormal{TV}(u)$ if and only if $p=\mathrm{D}^{*}q$ for some $q\in\partial\|\mathrm{D}u\|_{\mathcal{M}}$. Thus

[TABLE]

∎

Let now $u,v\in\textnormal{BV}(\Omega)$ such that $\mathcal{C}(\textnormal{D}u)\neq\emptyset$. From

[TABLE]

and our above observations we see, that the Bregman distance with respect to the TV functional penalizes the total variation of $v$ weighted by the deviation of image gradients (respectively their direction). Minimizing (2.5) hence favors a piecewise constant (denoised) image $v$ with a similar edge set (and hence structure) as $u$. Note that we again obtain no penalty for aligned gradient directions, and that it is in particular possible not to introduce an edge to $v$ even though $\mathrm{D}u$ indicates it. This feature reduces the risk of artificially introducing edges present in one image while the other is flat.

We give a brief example to further illustrate the idea. The distributional derivative of $u$ from the Sobolev space $W^{1,1}(\Omega)\subset\textnormal{BV}(\Omega)$ is given by $\mathrm{D}u=\nabla u\;\!\mathcal{L}$ with the Lebesgue measure $\;\!\mathcal{L}$. The associated polar decomposition is given by $f_{\mathrm{D}u}|\nabla u|\;\!\mathcal{L}$ with $f_{\mathrm{D}u}=\nabla u/|\nabla u|$ on the support of $\nabla u$. If $q\in\mathcal{C}(\textnormal{D}u)$ then $q=\nabla u/|\nabla u|$, $|\nabla u|\;\!\mathcal{L}$-a.e., and the Bregman distance between $u$ and $v\in W^{1,1}(\Omega)$ reads

[TABLE]

Minimizing the Bregman distance hence corresponds to weighted total variation, and favors an alignment of $\nabla v$ to $\nabla u$, $|\nabla u|\;\!\mathcal{L}$-a.e.

Since the Bregman distance is not symmetric, but the subgradients serve as one-sided a-priori information, we propose to solve the following iterative procedure for structural joint reconstruction:

[TABLE]

In each step of the procedure we solve the latter problem for one $u_{i}$, using the subgradients of the other $u_{j}$ as a-priori information, thus fitting the level sets of $u_{i}$ to the level sets of all $u_{j}$. Note that we have included the last iterate of $u_{i}$ as well, which guarantees the well-definedness of the problem and the usual properties of single channel Bregman iterations [10, 42]. The matrix $W=(w_{ij})\in\mathbb{R}^{N\times N}$ serves as a weighting between the different channels and is chosen such that its rows are normalized, i.e.

[TABLE]

Note that in case of $W$ being the identity we obtain decoupled standard Bregman iterations. The iteration scheme (2.6) is of course not limited to the total variation but can be carried out for arbitrary convex and absolutely one-homogeneous regularizers $J$. It has been proposed before by Moeller et al. [39] for color image processing and

[TABLE]

Starting with $u^{0}=0$ and $p^{0}=0$, the optimality condition of problem (2.6) allows us to compute new subgradients after having updated every $u_{i}$ once:

[TABLE]

Remark 2**.**

We mention that the set $\mathcal{C}(\mathrm{D}u_{i}^{k+1})$ can be empty for the solution $u_{i}^{k+1}$ of method (2.6), i.e. that there does not exist a continuous $q_{i}^{k+1}\in\partial\|\mathrm{D}u_{i}^{k+1}\|_{\mathcal{M}}$ such that $\mathrm{D}^{*}q_{i}^{k+1}=p_{i}^{k+1}$. It is important to notice that the above theory and Proposition 3 still hold in this situation, with the exception of the illustration (2.5). The derived method (2.6) solely depends on the $p_{i}^{k+1}$ from (2.8) and is hence well-defined also for $\mathcal{C}(\mathrm{D}u_{i}^{k+1})=\emptyset$. See also Remark 1.

With the subgradient updates (2.8) we shall briefly prove the well-definedness of the iteration scheme (2.6) by the direct method of the calculus of variations [22, Prop. 1.2]. Therefore, let $\mathcal{W}$ be the predual space of $\textnormal{BV}(\Omega)$, and assume that $K_{i}$ is the adjoint of a linear and bounded functional $L_{i}\colon Y_{i}^{*}\to\mathcal{W}$ on a Hilbert space $Y_{i}$ for every $i$.

Theorem 2**.**

Let the mapping $u_{i}\mapsto\mathcal{E}_{i}(u_{i})=\alpha_{i}H_{f_{i}}(K_{i}u_{i})+\textnormal{TV}(u_{i})$ be coercive and weak-∗ lower semicontinuous and $p_{j}^{k}$ be given by (2.8) for $k\geq 0$. Assume furthermore that the convex conjugate $H_{f_{i}}^{*}$ of $H_{f_{i}}$ is uniformly bounded and $w_{ii}>0$. Then there exists at least one solution to (2.6).

Proof.

For simplicity we let $\alpha_{i}=1$, and put iteration numbers in brackets in order to distinguish between powers and iteration numbers in this proof. We inductively show that

[TABLE]

for some $\zeta\in Y_{i}^{*}$. The crucial part is that due to (2.7)

[TABLE]

By the positivity of Bregman distances we have for $0<\delta<1$

[TABLE]

where $c_{1}=\min\{1-\delta,\delta w_{ii}^{k+1}\}$. Hence the mapping $u_{i}\mapsto\mathcal{E}_{i}(u_{i})$ is bounded on every sublevel set of the objective functional, and the Banach-Alaoglu theorem implies their precompactness in the weak-∗ topology. Since $p_{j}^{k}\in\mathcal{W}$ for all $j$, the mapping $u_{i}\mapsto\mathcal{E}(u_{i})$ is weak-∗ sequentially lower semicontinuous on bounded sets, which yields the assertion by a standard argument (see e.g. [22, Prop. 1.2]). ∎

Remark 3**.**

We like to further comment on the assumption $K_{i}=L_{i}^{*}$. Following the line of argumentation in [7], this assumption ensures that $K_{i}^{*}$ indeed maps $Y_{i}$ into $\mathcal{W}$ and not into the bigger space $\textnormal{BV}(\Omega)$. More precisely, this assumption implies that $K_{i}$ is sequentially continuous from the weak-∗ topology of $\textnormal{BV}(\Omega)$ to the weak(-∗) topology of $Y_{i}$ (note that they coincide on a Hilbert space) and hence possesses an adjoint $K_{i}^{*}$ mapping $Y_{i}$ to $\mathcal{W}$, regarded as a closed subspace of $\textnormal{BV}(\Omega)$. As a consequence, $p_{i}^{k+1}\in\mathcal{W}$, which is necessary to ensure lower semicontinuity in the weak-∗ topology of the linear term appearing in the Bregman distance.

Remark 4**.**

It is natural to ask the question about the convergence of the entire iterative procedure (2.6). In [39, Sec. 4.4 and 5] the authors discuss potential stationary solutions of the procedure and its convergence in some special cases. In case of identical operators $K_{i}=K$, symmetric weights $w_{ij}=w_{ji}$ and squared Hilbert space norms as data fidelities one can show that (2.6) converges for noiseless data $f_{i}$. With noisy data it is shown that the Bregman distance to the clean images decreases until the residual reaches the noise level, which is a generalization of the results for single channel Bregman iterations [42]. Unfortunately, little can be said about the general case, which at this point is subject of future research. The numerical studies however are encouraging, as they basically show the same behavior of (2.6) as single channel Bregman iterations even for very different imaging operators $K_{i}$ and asymmetric weightings.

2.3 Allowing anti-parallel vectors: Infimal convolution of Bregman distances

Taking a second glance at (2.4) or (2.5), we notice that the derived measure highly penalizes opposing vectors ($\cos(\varphi)=-1$) and forces them to point to the same direction, which requires that jumps across the edges of several images share the same sign. Or simply speaking, taking a “step up” across the edge between two structures in one image requires a “step up” in the other images, too. This can be suitable for some applications, e.g. for color image denoising and processing [39], since edges in different color channels of natural images most commonly seem to point to the same direction. However, this assumption is not always valid and in particular questionable in PET-MRI, so we adjust our method to allow opposing jumps across the edges. In view of (2.4) the most intuitive way is to apply the absolute value to the dot-product, i.e.

[TABLE]

Note that this resembles Ehrhardt et al.’s idea of parallel level sets (1.3) but bears the problem of non-convexity, and is hence not related to a Bregman setting. Instead we follow [39] and use the infimal convolution of two Bregman distances. The infimal convolution of two proper and convex functionals $J,L$ is defined as [3, p. 167]

[TABLE]

It is easy to show that the infimal convolution preserves convexity. It allows for a local decomposition of the argument $u$ into two parts, each trying to minimize one of the underlying functionals, while balancing the minimization of both. With the aim of losing the sign condition, it makes sense not only to look at the subgradient $q$, but also at the subgradient $-q$ with negative or “inverted” contrast. We recall that for absolutely one-homogeneous functionals $J$ we have

[TABLE]

so it is natural to look at the following infimal convolution of two $\|\cdot\|_{\mathcal{M}}$-Bregman distances.

Theorem 3**.**

Let $\nu,\eta\in\mathcal{M}(\Omega,\mathbb{R}^{m})$ such that $\mathcal{C}(\nu)\neq\emptyset$. Let $q\in\mathcal{C}(\nu)$. Then

[TABLE]

where

[TABLE]

$|\eta|$-a.e., and $\varphi$ denotes the angle between $f_{\eta}$ and $q$, i.e. $\cos(\varphi)|f_{\eta}||q|=f_{\eta}\cdot q$.

Since we already proved the point-wise case in [8] and the proof follows analogously, we have only included it in the Appendix. At first we recognize that $\mathrm{ICB}_{\|\cdot\|_{\mathcal{M}}}$ behaves like (2.9) on one part of its domain, and that we are in particular losing the sign of $q$ everywhere. Furthermore, since the Bregman distance is convex in its first argument if we fix the second, this functional is convex. For a detailed description of the behavior, we at first pass over to the gradient setting again, i.e. for $q\in\mathcal{C}(\textnormal{D}u)$

[TABLE]

We again distinguish two different situations: either $u$ has an edge, which corresponds to a non-zero (distributional) gradient, or $u$ is constant with vanishing gradient. In the former we find that $|q|=|f_{\mathrm{D}u}|=1$, which is only possible in the “else”-case of (2.10), and the functional vanishes. This results in no penalization of $\mathrm{D}v$, which allows to introduce an edge of arbitrary height to $v$ for free, hence again locating edges at the same positions. We observe that here the direction of the edge is arbitrary and refer the reader to [8] for some further elaboration on that. In case it is constant, $u$ does not offer any structural information, but it is desirable to nevertheless have a regularizing effect on $v$ or $\mathrm{D}v$, respectively. However, the interpretation is a bit more delicate in that case. The subgradient $q$ here is a vector from the unit ball with length $|q|\leq 1$.

Inserting the conditions into the integrand of $\eqref{eq:IC_integrand}$ yields an upper bound for the value of the integrand in the first case, and a lower bound for the second case. For the first case and $|q|<|\cos(\varphi)|$ we have

[TABLE]

For the second case we find

[TABLE]

and thus for the value of the integrand

[TABLE]

We combine both results and obtain

[TABLE]

So in order to minimize the functional it is beneficial to either assign no mass to $\mathrm{D}v$ or to find a combination of $f_{\mathrm{D}v}$ and $q$ such that $|q|<|\cos(\varphi)|$. Figure 1 illustrates the situation for different values of $|q|$. The larger the magnitude of $q$, the narrower is the range of vectors around $q$ that we can pick to fulfill $|q|<|\cos(\varphi)|$. Hence the functional tries to align $f_{\mathrm{D}v}$ to $q$ to obtain the smallest penalty. We again find that $|q|$ can serve as a probability of how likely an edge is at that position, while its direction implies a range for the direction of the jump which narrows with increasing size of $q$.

Proposition 4**.**

Let $u,v\in\textnormal{BV}(\Omega)$ with derivatives $\mathrm{D}u,\mathrm{D}v\in\mathcal{M}(\Omega,\mathbb{R}^{m})$, and $p\in\partial\textnormal{TV}(u)$. Then

[TABLE]

respectively

[TABLE]

for some $q\in\partial\|\mathrm{D}u\|_{\mathcal{M}}$.

Proof.

For $w=0$ and $w=v$ we have

[TABLE]

which yields the first inequality. Then by definition we have

[TABLE]

∎

As a consequence, minimizing $\mathrm{ICB}_{\mathrm{TV}}^{p}(u,v)$ immediately implies a small $\mathrm{ICB}_{\|\cdot\|_{\mathcal{M}}}^{q}(\mathrm{D}v,\mathrm{D}u)$ and hence the desired behavior for joint reconstruction. Following [39], we propose the following iteration scheme:

[TABLE]

Similarly to (2.6), we solve the latter problem for one $u_{i}$ in each step, using the subgradients of the other $u_{j},j\neq i$ as a-priori information. We hence fit the edge set of $u_{i}^{k+1}$ to the edge sets of all $u_{j}^{k}$, now excluding the direction/sign constraint via the infimal convolution. We emphasize the use of the usual Bregman distance on the diagonal, which is the obvious choice, since in contrast to all the foreign channels the sign should be relevant on the own channel.

Starting with no prior knowledge about subgradients, i.e. $p_{i}^{0}=0$ for all $i=1,\dots,N$, the first results $u_{i}^{1}$ are separate TV-reconstructions without any coupling. We then update the subgradients for the first time and gain coupled reconstructions for all subsequent iterations. The general procedure is similar to the use of usual Bregman iterations [42, 10]: Starting with a very high regularization ($\alpha_{i}$ small), the first iterates are very smooth and only contain large scales. Hence the subgradients essentially contain information about the edges of large image features. Since for Bregman iterations the regularization behaves proportional to $\frac{1}{\alpha_{i}k}$, the amount of regularization decreases with every iteration and we gain more structure in the subgradients, i.e. a better knowledge about edge locations of smaller features. The iteration has to be stopped when noise reappears in the images. The ratio between the parameters $\alpha_{i}$ is crucial for the outcome of the joint reconstruction, as we have to ensure that the reconstructions evolve equally fast.

Remark 5**.**

Unfortunately, we cannot immediately obtain the necessary subgradient updates from the optimality condition of the problem due to the involved infimal convolution. Without an explicit structure of the subgradient (cf. (2.8)) it is in particular impossible to prove the well-definedness of the minimization problems in a similar manner as Theorem 2, since there is no particular reason for the problem to be coercive for an arbitrary subgradient $p_{i}$. For the same reason it is hard to characterize a potential convergence of procedure (2.12) (cf. Remark 4). However, for the numerical realization of the method, we shall find an alternative, numerical update for the subgradients using a primal-dual scheme.

We already pointed out that both methods (2.6) and (2.12) naturally exclude the size of image gradients. Interestingly, they additionally show a certain kind of invariance under positive rescaling of the data $f_{i}$.

Proposition 5**.**

For any $c>0$ let $H_{f_{i}}$ satisfy $H_{cf_{i}}(cu_{i})=c^{r}H_{f_{i}}(u_{i})$ for some $r\geq 1$. Furthermore, for fixed $\alpha_{i}$ and $f_{i}$, denote the solution of (2.6) respectively (2.12) by $u_{i}^{k+1}$. Then the solution of (2.6) respectively (2.12) for any rescaling $\tilde{f}_{i}=c_{i}f_{i}$ of the data $f_{i}$ by $c_{i}>0$ and regularization parameter $\tilde{\alpha}_{i}=\alpha_{i}/c_{i}^{r-1}$ is given by $\tilde{u}_{i}^{k+1}=c_{i}u_{i}^{k+1}$.

Proof.

By the absolute one-homogeneity of TV we have that

[TABLE]

i.e. both functionals are positively one-homogeneous. We immediately conclude that

[TABLE]

The proof for (2.12) follows analogously. ∎

We mention that the assumptions on the data terms $H_{f_{i}}$ are easy to verify for quadratic data terms and the Kullback-Leibler divergence.

2.4 One-dimensional illustration

As a first proof of concept, let us have a look at a one-dimensional example with two channels, featuring all the common issues of a typical joint reconstruction setting. Figure 2(a) shows two piecewise constant signals, whose edges are located at the exact same positions, namely $10,30,50,70$ and $90$. However, only some of the jumps across the edges share the same direction, which is either a “jump up” or a “jump down” in both channels, while the others have opposite direction. The former case occurs at positions $30$, $70$ and $90$, the latter at positions $10$ and $50$. We would like to put special emphasis on the varying height of jumps in both channels. Since both signals feature small and large jumps at different positions, an overall scaling of the signals in order to have an approximately equal height of jumps everywhere is not possible. For example, scaling the first jump at position $10$ to equal height requires to “shrink” the red signal, which however leads to unreasonably large differences e.g. at location $70$.

Both signals have been artificially corrupted by additive Gaussian noise with zero mean and standard deviation $\sigma=0.35$, which can be seen in Figure 2(b) and (c). Undoubtedly it is basically impossible to recover both signals without any further knowledge, since especially in the blue channel the noise covers the two small jumps of the signal. But since we know that the edges are located at the same positions, we can apply our method to couple the edge positions during the reconstruction. Due to the Gaussian noise, the associated joint reconstruction method is:

[TABLE]

with weights $\lambda,\mu\in[0,1]$. Note that in one dimension we indeed only couple the edge positions, since the “direction” of jumps across the edges is limited to “up” or “down”, which is however excluded by the infimal convolution. Without any doubt an ordinary TV reconstruction would not be able to recover the original signal from the noisy data. Hence, as a comparison to the joint reconstruction result we provide individual reconstructions of both channels with a Bregman TV prior, which corresponds to $\lambda,\mu=1$ and decouples the channels. Since we iteratively include edge information at least from the own channel, chances are higher to find the right location of edges and the right height of jumps. For both separate and joint reconstructions, the crucial question is how many Bregman iterations to perform. For the former we chose to iterate until all edges that are present in the original signal became visible. The result can be found in Figure 3(a).

Though the denoising was successful and the individual appearance of the signals is acceptable, a large part of the edges is located at wrong positions and even some new edges are introduced to the signal. Even worse, comparing between channels, the assumption of equal edge positions is not met. In contrast, the joint reconstruction result (b) after $7$ Bregman iterations shows a perfect recovery of the edge sets of both signals. With regard to the intensities of the signal, or the height of the jumps, we obtain slightly different values than for the original signal. This is however not surprising, since the mean of this specific noise realization is not exactly zero. And indeed, taking the mean value of the noisy signals between the (known) edge locations in Figure 3(c) reveals the exact same signal as the joint reconstruction, which confirms that the joint reconstruction is a perfect recovery of the data over the true edge set.

A closer look at the infimal convolution delivers further insight into its behavior. Recall that we have e.g. for the blue channel

[TABLE]

Here we seek for a minimizing decomposition of the blue signal into two parts, such that one part matches the subgradient $p_{\mathrm{red}}^{k}$ and the other part matches the negative subgradient $-p_{\mathrm{red}}^{k}$. To do so, $z_{\mathrm{blue}}$ has to compensate for jumps at positions where the two channels do not share the direction of the jump, i.e. at locations $10$ and $50$. Figure 4(a) shows the first part of the decomposition for both channels, namely $\mathbf{u}-\mathbf{z}$. Intuitively, the signal $\mathbf{u}-\mathbf{z}$ should only have jumps at locations where the signals share edges with the same direction and otherwise be constant, and indeed this is the case. In contrast, the variable $\mathbf{z}$ itself has to match the negative subgradient of the other channel, i.e. it has to have the same jump locations and directions as the negative signal from the other channel. This can be seen in Figure 4(b) and (c). Note that the signals do not have to share every edge, but also constant signals are possible which can be seen e.g. in (b) at location $30$.

We continue by showing how to solve the joint reconstruction problem numerically, before we get to a more sophisticated numerical example, namely PET-MRI joint reconstruction.

3 Numerical solution

In this section we show how to solve the joint reconstruction problem numerically. Due to the non-differentiability of the involved Bregman distances we employ a primal-dual method (cf. e.g. [44, 23, 13]). In general, the idea is to dualize every term of (2.12) containing an operator using the notion of convex conjugates [22]

[TABLE]

The goal is to obtain a saddle-point problem such that the proximal maps

[TABLE]

for all the involved functions are easy to solve via point-wise operations or simple projections. This is e.g. the case for the characteristic function $\delta_{S}$ of a convex set $S$, i.e.

[TABLE]

where

[TABLE]

The advantage of the saddle-point formulation is that the proximal maps of dualized terms decouple, which allows us to treat every Bregman distance separately. Since every additional channel simply introduces another infimal convolution we restrict ourselves to two channels and comment on the straight-forward extension to an arbitrary number of channels at the end of the section. We nevertheless use a notation with indices $i$ and $j$ rather than $1$ and $2$, in order to unify the derivation for later use and to make the notation as simple as possible. The channel which is currently reconstructed is referred to by $i$, while the foreign channel related to the infimal convolution is indexed by $j$. So let us consider two inverse problems

[TABLE]

$u_{i}\in\mathbb{R}^{N}$, $f_{i}\in\mathbb{C}^{M_{i}}$, $K_{i}\in\mathbb{R}^{M_{i}\times N}$ for $i=1,2$, that we want to solve jointly for $u_{i}\in C_{i}$, where $C_{i}\subset\mathbb{R}^{N}$ are convex sets. We discretize the two dimensional gradient $\nabla\colon\mathbb{R}^{N}\to\mathbb{R}^{N\times 2}$ using standard forward differences (see e.g. [13]) and define the (isotropic) total variation of $u\in\mathbb{R}^{N}$ as

[TABLE]

We make use of the specific structure of the subdifferential of TV. By the chain rule we find that $p\in\partial\textnormal{TV}(u)$ if and only if there exists $q\in\partial\|\cdot\|_{1}(\nabla u)$ such that $p=-\nabla\cdot q$. Hence, let us first assume that we are already given a subgradient

[TABLE]

of the current approximations $u_{i}^{k}$. The next step of the method is then to solve

[TABLE]

In order to derive the saddle-point formulation of the problem we have to dualize every involved term except for the characteristic function. Recall that we cannot access a general closed form representation of $\mathrm{ICB}_{\mathrm{TV}}$.

Instead, we use the definition of the infimal convolution to introduce an additional auxiliary primal variable $z$ into the problem:

[TABLE]

We observe that the subproblem we have to solve in every iteration basically consists of only two parts: A (differentiable) data term and a sum of total variation Bregman distances with regard to different subgradients. Due to the structure of their subgradients, the Bregman distances can be dualized as a shifted and scaled $\ell_{1}$-norm, which we show at the example of the first arising Bregman distance

[TABLE]

Here, the set $S$ is defined as

[TABLE]

The proximal map for the characteristic function $\delta_{S}$ can be computed explicitly as a projection onto the set $S$, or a shifted projection onto the set $\tilde{S}=\{s~{}|~{}\|s\|_{\infty}\leq w_{i}\}$:

[TABLE]

Proceeding analogously with the two other Bregman distances and dualizing the data term we end up with the following primal-dual formulation:

[TABLE]

where

[TABLE]

Hence, all the proximal updates for the involved Bregman distances can be carried out as a projection on one of the sets $S_{l}$. We illustrate one step of the numerical algorithm in Algorithm 1 and summarize the whole reconstruction process in Algorithm 2. In order to avoid too many indices, we drop the Bregman iteration number $k$ and still focus on only one channel. The other one follows analogously by simply exchanging the roles of $i$ and $j$.

The extension to more than two channels is then easily done. Since an additional channel adds a second infimal convolution to the problem, we can introduce another auxiliary primal variable to obtain two additional Bregman distances. These can be treated as before by a projection onto the sets associated to the involved subgradients.

3.1 Stopping criteria

For appropriate results it is necessary to decide when to consider the single subproblems (3.1) as converged. We use a modified primal-dual gap in the following sense: the dual problem of (3.1) (respectively (3.2)) is given by

[TABLE]

We hence define a primal-dual gap for the $k$-th Bregman iteration of the $i$-th channel as

[TABLE]

which has to converge to zero. Note that due to the projection onto $S_{j}$ in every step of Algorithm 1, the constraint on $y_{j}$ for $j=2,3,4$ is trivially fulfilled. The latter two constraints however have to be checked separately alongside the gap. We relaxed the third constraint and normalized the gap by the number of primal pixels $N$ such that we consider the subproblem as converged if

[TABLE]

We remark that the evaluation of the gap does not require any severe computational costs since the application of the imaging operator $K_{i}$ and the gradients have to be done for the next iteration anyway. It is also worth mentioning that in practice the second constraint seemed to be the best measure of convergence, since it was usually fulfilled last. Interestingly it can even serve as an indicator, which pixels in $u$ have not converged yet, since they directly correspond to the positions not fulfilling the second constraint.

3.2 Subgradient updates

After convergence of both channels, we need to update the subgradient for the next Bregman iteration. Unfortunately we cannot find an update equation equivalent to (2.8) directly from the optimality condition of the problem due to the involved infimal convolution. But since we can access the dual variables, we can derive an alternative subgradient update. Note that for a stationary point $((u,z),(y_{1},y_{2},y_{3},y_{4}))$ of problem (3.2) the second dual variable $y_{2}$ needs to fulfill

[TABLE]

Recall that $\delta_{S_{2}}$ is the convex conjugate of a Bregman distance which is proper, convex and lower semicontinuous. By duality, the above inclusion is hence equivalent to

[TABLE]

which yields the subgradient update

[TABLE]

It is important to notice that the convergence of the algorithm obviously influences the quality of the obtained subgradient. In particular it can happen that a poor approximation does not even fulfill the requirements of a subgradient (2.2), i.e. $|q_{i}^{k+1}|_{l}>1$ for some index $l$. This can be avoided by a sufficient convergence of the subproblem, or can even be manually corrected for if necessary. With the tolerances chosen in our numerical studies however we did not experience any problems.

4 PET-MR imaging and numerical results

In the following we give some further motivation for the use of structural joint reconstruction using a topic of very recent interest, namely PET-MR imaging, which is the example for the numerical studies at the end of this section. Similar studies can be found e.g. in [21, 33].

4.1 PET-MRI modeling

While widely established imaging techniques such as PET-CT only allow for a sequential data acquisition, the coupling of PET and MR imaging devices offers further opportunities of mathematical reconstruction and finally clinical diagnosis. Since these scanners are able to simultaneously acquire functional PET data and structural MR data they provide access to spatially and temporally registered data of two complementing imaging techniques. PET imaging is capable of showing metabolic processes and is therefore e.g. useful to locate tumors and diseased areas of the heart, but unfortunately suffers from a poor resolution and a lack of anatomical information. In contrast, MR imaging yields a very high resolution with great contrast between different types of tissue, but provides only little possibilities to display metabolic processes and realize quantitative imaging. Additionally, it is common practice to speed up the data acquisition for MRI by undersampling the $k$-space, i.e. by measuring only a fraction of the data, which degrades the quality of the resulting reconstruction. However, having data from both modalities at hand, one can try to exploit the additional information in order to re-improve the results, in particular for PET, but as well for MRI. Since we cannot expect function to be independent of the underlying anatomy [21], it is reasonable to assume a similar structure in both the PET and the MR image, which motivates a structural joint reconstruction.

We briefly introduce the problem of PET and MR imaging and establish our notation, restricting ourselves to a setting in two dimensions. For a more sophisticated introduction we refer the reader e.g. to [16, 52] for PET and [36, 49] for MRI.

PET makes use of the radioactive decay of an artificially created positron emitter, so-called tracer, applied to the patient’s body. The emitted positrons annihilate with free electrons from the surrounding tissue, thereby generating two gamma photons which travel in (almost) opposite direction. The pair of gamma photons can be measured by a detector ring surrounding the patient. If two gamma photons impact on the detector surface at approximately the same time, they are likely to originate from the same decay. From the positions of the impacts we thus gain information about the approximate location of the decay, which must have taken place along the connecting line of response. The collectivity of all lines of response mathematically corresponds to a fan-beam-type sampling of the Radon transform (cf. e.g. [40]) in two dimensions, i.e. line integrals over the spatial tracer distribution between two detector pairs.

Hence let $\Omega\subset\mathbb{R}^{2}$ be an open and bounded image domain and $u\colon\Omega\to[0,\infty)$ denote the tracer distribution. The imaging process is usually modeled as a semi-discrete operator equation, mapping the continuous representation of $u\in L^{1}(\Omega)$ to the discrete data $f\in\mathbb{R}^{M}$ [46, 47]. We restrict ourselves to the fully discrete case, so the imaging operator is a matrix $A\colon\mathbb{R}^{N}\to\mathbb{R}^{M}$ mapping $u\in\mathbb{R}^{N}$ on a finite grid $\Omega_{0}$ to the discrete data $f$. We aim to solve the inverse problem

[TABLE]

This is essentially an integral operator equation, and as the data always contains noise we are in need of a-priori information to stabilize the reconstruction process. In general, the underlying random process of tracer decay and photon detection as well as noise is assumed to be Poisson distributed. Thus the reconstruction of $u$ is commonly carried out as the maximum a-posteriori estimator for $u$, which leads to the minimization problem

[TABLE]

where we used a Gibbs prior $p(u)=\exp(-\alpha^{-1}\mathcal{R}(u))$ for the a-priori density of $u$. By adding the term $-f+f\log(f)$ (we use the convention $0\log(0):=0$) independent of $u$ we gain the famous Kullback-Leibler divergence, which we denote by $\textnormal{D}_{\textnormal{KL}}(f,Au)$. Note that the value of $\textnormal{D}_{\textnormal{KL}}(f,Au)$ has to be set to infinity if $(Au)_{i}=0$ for some $i\in\{1,\dots,M\}$. Since $u$ represents a density, it is common practice to add a positivity constraint on $u$. For problem (4.2) to be well-posed it is usually necessary to add some additional technical assumptions on the imaging operator $A$. We require the operator $A$ to preserve positivity, i.e. if $u\geq 0$ then necessarily $Au\geq 0$, and refer the reader to [47, p. 79] for a deeper discussion of the topic.

The choice of the prior $\mathcal{R}$ now determines the type of a-priori knowledge we add to the problem. Later, we compare our joint reconstruction method to individual reconstructions using no prior (Expectation Maximization with early stopping), Total Variation regularization (TV, see e.g. [45, 47, 4, 30]), as well as to joint reconstruction techniques via Joint Total Variation (JTV [28]) and Parallel Level Sets (PLS [21]).

MR imaging makes use of the nuclear spins of hydrogen nuclei inside the human body. By applying several magnetic fields to the patient, these spins can be specifically aligned such that one can measure differences in the resulting resonant frequencies. Mathematically, the recorded data sets correspond to measurements of the sought-for quantity in Fourier space [36]. For simplicity, we ignore the phase of the MR image and restrict our studies to absolute magnitude imaging [9]. Thus the MR image is a non-negative, real-valued image $v\colon\Omega\to[0,\infty)$. As for PET, we restrict ourselves to the discrete case for the modeling. Hence, the imaging operator $B$ is essentially a discrete Fourier transform which maps the discrete $v\in\mathbb{R}^{N}$ to the Fourier measurements $g\in\mathbb{C}^{L}$ and we aim to solve

[TABLE]

Assuming the noise in MRI to be Gaussian, we can seek for $v$ as the maximum a-posteriori estimator, i.e.

[TABLE]

We shall denote the data term by $\|\cdot\|_{\mathbb{C}}^{2}$. It is common practice to accelerate the MR data acquisition by measuring only a sparse set of Fourier coefficients. The missing information is then compensated for by suitable a-priori knowledge. The most popular approach is compressed sensing ([11, 12, 18, 38], where the main idea is to map the MR image to some function space where it has a sparse representation, and can thus be reconstructed from substantially reduced data. The undersampled frequencies are usually measured randomly or, for practical reasons, on a geometric pattern such as spirals or spokes, which causes artifacts in the reconstruction. As for PET, we compare the results of our joint reconstruction to already existing individual and joint reconstruction techniques. These are a zero-filled inverse Fourier transform (no prior), a TV-prior for sparsity in the gradient domain (TV), Joint Total Variation (JTV) as well as a joint PLS reconstruction (PLS).

4.2 Phantoms and data sets

For the numerical studies we set up an artificial brain phantom (see Figure 5) with a size of $272$x$272$ pixels using data from BrainWeb [15], which features the typical challenges of a joint reconstruction.

First of all, both phantoms feature the exact same location of edges everywhere except for the two artificially added lesions in the upper right part of the PET image and the left part of the MR image. Both have been added to illustrate a common joint reconstruction problem, namely the introduction of artifacts, i.e. features not present in one of the images that are transferred artificially to the other. While the locations of the edges are equal, the jumps across the edges are not, meaning that we e.g. have a “step up” from gray matter to white matter in the MR image, while we have a “step down” in the PET image. Furthermore, we point out the different range of image intensities. This imposes the issue of locally different gradient heights, which cannot be dealt with thoroughly by a global scaling of the data. Summing up we find the following features:

• •

Equal edge locations (except for two lesions),

• •

Equal and different edge orientations,

• •

Different height of jumps across edges,

• •

Different scale.

The phantoms have been used to create artificial data sets for PET and MR imaging. The PET operator is taken from EMrecon [34] and is modeled to match the geometry of one detector ring of the Siemens mMR scanner. It features 56 transversal detector blocks at 8 crystals each, and the gaps between detector blocks have been artificially filled by 56 virtual crystals. For further details we refer the reader to [17]. In order to model the lack of resolution, positron range and scatter in PET, we employ a 2-dimensional Gaussian blur in the image domain with a full width at half maximum (FWHM) of $4$x$4$mm, where we assume that the pixel size in the image is $1$mm. The blurring kernel thus corresponds to a Gaussian with a standard deviation of $\sigma=1.7$. The PET data is an instance of a Poisson distribution, where we simulated a total number of $1.5$ million counts. However, we point out that for artificial data the total number of counts is a rather poor criterion to judge a data set, since its quality rather depends on the distribution of those counts across the phantom.

In case of MRI we use undersampled $k$-space data, meaning that we sample the Fourier space only at a few frequencies specified by different geometries. For the sake of simplicity, the MRI operator hence consists of a 2-dimensional Fourier transform followed by a projection onto the geometric pattern of the corresponding sampling (cf. [20] . Note that we do not use a non-uniform Fourier transform since the chosen frequencies are still located on a Cartesian grid. However, the idea and method do not change for non-Cartesian methods. Eventually, Gaussian noise with an energy of approximately five percent of the total energy of the data set is added. We show four different types of undersampling in this work, which can be seen in Figure 6. The samplings introduce different types of artifacts and hence serve different purposes for the joint reconstruction setting which we elaborate on alongside with the results below.

4.3 Methods

We compare our method (ICB)

[TABLE]

to several reconstruction techniques: separate reconstructions with no prior (Expectation Maximization [48] with early stopping for PET and zero-filled inverse FT for MRI) and a total variation prior, as well as joint reconstructions via joint TV (JTV [28]) and linear parallel level sets (LPLS [21]). Since decoupled Bregman iterations did not provide a substantial improvement over TV, we did not include these results. For JTV and LPLS we used the implementation provided by the authors of [21]. We mention that due to the nonconvexity of the LPLS method we follow [21] and use the result of JTV as the initialization for quadratic parallel level sets (QPLS [21]), and the result of QPLS to initialize LPLS. Since the QPLS reconstructions have always been inferior to LPLS we left them out of this paper as well. Hence, for separate PET reconstructions we solve

[TABLE]

and accordingly for MRI

[TABLE]

The joint reconstruction problems read

[TABLE]

where

[TABLE]

All parameters were tuned to maximize the self-similarity index (SSIM [51]) between the reconstructions and the ground truth over the region of interest, where for joint reconstruction methods we maximized the ‘joint’ SSIM, i.e. the sum of both SSIM values. For all joint reconstruction techniques, the ratio between the parameters $\alpha$ and $\beta$ has been chosen such that the size of the data terms is approximately equal. This guarantees an equal regularization for both the PET and the MR image, which corresponds to a uniform evolution of both the PET and MR image for ICB in terms of regularity during the Bregman iterations. We mention that there is potential further improvement by tweaking this ratio for data of different quality. The Bregman iterations for ICB have been stopped when noise reappeared in the images. Starting with high regularization, we performed 50 Bregman iterations on average. With the numerical implementation outlined in Section 3, $tol_{gap}=1e-4$ and $tol_{constraint}=1e-5$, every Bregman iteration took approximately three minutes on a standard laptop, which amounts to a total computation time of approximately three hours. We however remark that it is possible to reduce both the amount of Bregman iterations and the tolerances without substantially reducing the quality of the results. The SSIM values can be found in Table 1 alongside with the weighting between channels for ICB.

4.4 Results

Figures 9 to 12 show the results for four different MRI samplings and different types of regularization. The left two columns display the PET and MRI reconstructions, where all pictures are put onto the original scale of the ground truth, i.e. $[0,10]$ for PET and $[0,1]$ for MRI. If the reconstructions overestimate the image values beyond that scale, the corresponding pixels are set to the maximum value (note that an underestimation below zero is impossible due to the positivity constraint). The effective over- or underestimation of image values with respect to the ground truth can be assessed from the difference images on the right-hand side of the figures. Note that due to the missing weighting between gradients for JTV and LPLS, we had to rescale the PET data by a factor of $10$ in order to approximately provide the same range of image intensities for both PET and MRI. The results have then eventually been rescaled to their original range. We mention that due to the nonlinearity of the methods this may result in a slight change of quantitative values.

4.4.1 Full sampling

For a full sampling, already a separate reconstruction provides a visually perfect MR image, which remains the case for all priors. Here, the weighting for ICB is chosen such that the PET image does not influence the reconstruction of the MR image ($\mu=1$) and the procedure is similar to a PET reconstruction with an (evolving) anatomical prior. The MR reconstruction performed in parallel hence corresponds to a single channel BTV reconstruction. The quality of the PET image varies greatly. Without any prior, the PET image shows the typical noisy and blurry appearance of reconstructions from noisy Poisson data. The TV prior removes the noise, but highly oversmoothes the image. The first joint reconstruction method, JTV, is able to transfer some part of the sharp structures of the MR image to the PET image and increases its quality. However, the result remains too smooth. LPLS and ICB both show a substantial improvement of the image quality in terms of sharp edges and noise reduction. However, LPLS still features some remaining noise, and the lesion in the MRI not present in PET has been partly transferred. The result for ICB seems visually perfect on the shared structures of the image which can as well be assessed from the difference image. The only drawback is the smoothing of the hot lesion only present in the PET image, while we however observe that the MRI lesion has not been transferred. The observations are as well confirmed by the SSIM values in Table 1.

The advantage of ICB over LPLS is easily explained by the weighting between channels. While LPLS always relies on a 50/50 weighting, the result for ICB has been obtained with an MR influence of $90\%$. For further comparison to a 50/50 weighting (and the overall influence of the channel weighting) we provide a weighting series in Figure 7.

4.4.2 Half sampling

Sampling every second line of the k-space introduces a ghosting artifact into the MR image. JTV and LPLS show a similar performance in removing some parts of these artifacts from the MRI, while LPLS as well shows a clear improvement of the PET image. However, both lesions are again artificially shared between the images. The ICB results contain the least artifacts for MRI and the highest improvement for the PET image, where however again the hot lesion is attenuated. We as well find a slight transfer of the lesion from the MR image into the PET image, whereas the lesion from PET is not transferred to the MR image.

4.4.3 Spokes sampling

The situation for a spokes sampling is almost identical to a full MR sampling. Since already a TV prior removes the resulting grain artifacts, we again choose the weighting for ICB such that the PET image does not influence the MR image ($\mu=1$). Interestingly, JTV and LPLS decrease the quality of the MR image, both visually and in terms of SSIM, by re-introducing some part of the noise. However, the PET image still benefits significantly from the influence of the MRI. ICB again shows the best performance for PET, sharpening the edges and restoring the quantitative values, where we again observe the attenuation of the hot lesion and transfer of the MR lesion.

4.4.4 Spiral sampling

For a spiral sampling, we again expect a mutual benefit for both modalities, since the sampling introduces some circular artifacts. Surprisingly, JTV delivers the best result in terms of SSIM, which however look too smooth. Visually the results for LPLS convince the most, both for PET and MRI. The ICB result for PET is sharper and more accurate in terms of quantitative values, but appears a little too noisy, which can as well be caused by the influence of the undersampled MR image.

4.5 The benefit of channel weighting

We further comment on the channel weighting, i.e. the parameters $\lambda$ and $\mu$. As already mentioned in the previous section, a known drawback of joint reconstruction methods is a cross-talk between the different channels, i.e. channels might cause artifacts in other channels. This can e.g. be the transfer of a structure from one channel to the other (cf. the PET image in Fig. 11), or the smoothing out of structures since they are not present in all channels (cf. the lesion in the PET image in Fig. 9). This is an issue which is hard to avoid. However, the weighting between channels, i.e. the parameters $\lambda$ and $\mu$ can indicate, which structures are likely to be artifacts or falsely influenced by the joint reconstruction. Figure 7 shows the PET reconstruction with an increasing amount of influence of the MR channel. In comparison to a seperate reconstruction (upper left), already an influence of 10 percent of the MR image substantially sharpens the PET image. This trend continues as the influence further increases. Unfortunately, the hot lesion in the upper right part of the PET image starts to be smoothed out with increasing MR weighting. This error, however, can be identified by a look at the seperate reconstruction and low MR weightings.

This of course works in both ways. Figure 8 shows the MR image for the ‘half’ sampling under increasing weighting of the PET image. Already an influence of 10 percent substantially reduces the ghosting artifacts, which however cannot be suppressed entirely even for higher weighting.

Hence, a series of reconstructions with different channel weightings can help identify artifacts caused by the joint reconstruction.

5 Conclusion

We extended the ideas of Moeller et al. [39] for color image processing to a more general joint reconstruction setting, involving data and hence images of totally different types. The presented method is able to exploit structural similarities between images during the reconstruction to obtain significant improvements even for highly degraded data. We gave insights and justification for the method in a rigorous, theoretical setting and extended the idea from a discrete vector space setting to the space of finite Radon measures.

Using PET-MR imaging as a particular numerical example, we demonstrated that both modalities can substantially benefit from a joint reconstruction. We furthermore compared the method to already existing joint reconstruction methods and showed a similar or in many cases superior performance.

There are a few questions remaining open for the future. First of all, a current subject of investigation is the choice of subgradients, which still leaves room for improvements. This could in particular solve the missing existence proof for the infimal convolution method. Regarding applications, an investigation of the method’s performance on real data is certainly interesting, as well as the transfer to other fields such as functional MRI, which is subject of current research. Last but not least, we are investigating other numerical approaches to reduce the computational effort for the solution of the method.

Appendix A Appendix

A.1 Proof of Theorem 3

Theorem 4**.**

Let $\nu,\eta\in\mathcal{M}(\Omega,\mathbb{R}^{m})$ such that $\mathcal{C}(\nu):=\partial\|\nu\|_{\mathcal{M}}\cap C_{0}(\Omega,\mathbb{R}^{m})\neq\emptyset$. Let $q\in\mathcal{C}(\nu)$. Then

[TABLE]

where

[TABLE]

$|\eta|$-a.e., and $\varphi$ denotes the angle between $f_{\eta}$ and $q$, i.e. $\cos(\varphi)|f_{\eta}||q|=f_{\eta}\cdot q$, with $\varphi(x):=0$ if $q(x)=0$.

Proof.

We can adapt the idea of the proof in [8]. Let

[TABLE]

It follows from $(f_{1}\Box f_{2})^{*}=f_{1}^{*}+f_{2}^{*}$ and the definition of the biconjugate that

[TABLE]

We start with computing the right-hand side and find $f_{1}^{*}(w)=\iota_{\mathcal{M}^{*}}(w+q)$ and $f_{2}^{*}(w)=\iota_{\mathcal{M}^{*}}(w-q)$, where $\iota_{\mathcal{M}^{*}}$ denotes the characteristic function of the dual ball

[TABLE]

Since $C_{0}(\Omega,\mathbb{R}^{m})\subset\mathcal{M}(\Omega,\mathbb{R}^{m})^{*}$, we deduce

[TABLE]

Due to the constraints, we can carry out the computation pointwise (respectively $|\eta|$-a.e.). We first address the case $|q|=1$, which immediately implies that $w=0$ and $w\cdot f_{\eta}=0$. Hence we can from now on assume that $|q|<1$ and set up the corresponding Lagrangian for $|\eta|$-a.e. $x\in\Omega$

[TABLE]

where we leave out the dependence on $x\in\Omega$ for simplicity. Note that $w,q,f_{\eta}\in\mathbb{R}^{m}$ and that we minimize the Lagrangian here. Every optimal point of (A.2) has to fulfill the four Karush-Kuhn-Tucker conditions, namely

[TABLE]

and Slater’s condition implies the existence of Lagrange multipliers for a KKT-point of (A.2). The first KKT-condition yields

[TABLE]

We note that the case $f_{\eta}=0$ (which can happen on $|\eta|$-zero sets) causes the objective functional to vanish, hence in the following $f_{\eta}\neq 0$.

Let us first consider the case $q=0$ in which (A.3) yields $f_{\eta}=2(\lambda_{1}+\lambda_{2})w$. In case $|w|<1$, we find that $(\lambda_{1}+\lambda_{2})=0$ and hence $f_{\eta}=0$, which is a contradiction. In case $|w|=1$, we obtain that $(\lambda_{1}+\lambda_{2})=\frac{1}{2}$ and $w=f_{\eta}$ from (A.3), hence

[TABLE]

If $q\neq 0$, we have to distinguish three cases:

1st case: $|w-q|<1,|w+q|=1$.

Thus $\lambda_{1}=0$ and (A.3) yields $f_{\eta}=2\lambda_{2}(w+q)$. Since $|w+q|=1$ and $|f_{\eta}|=1$ we deduce $\lambda_{2}=\frac{1}{2}$, so $w=f_{\eta}-q$ and finally for the value of the objective function

[TABLE]

2nd case: $|w+q|<1,|w-q|=1$

We analogously find

[TABLE]

The first two cases thus occur whenever (insert $w$ in the conditions) $\left|f_{\eta}\pm 2q\right|<1$, meaning that either of the conditions has to be fulfilled. We calculate

[TABLE]

Hence $q\cdot f_{\eta}>0$ and

[TABLE]

In the second case we analogously find $|q|<-\cos(\varphi)$, hence $q\cdot f_{\eta}<0$ and

[TABLE]

so we may summarize the first two cases as

[TABLE]

if $|q|<|\cos(\varphi)|$.

3rd case: $|w-q|=1,|w+q|=1$

At first we observe that from $|w+q|^{2}=|w-q|^{2}$ we may deduce that $w\cdot q=0$. Therefore we have $|w+q|=1\Rightarrow|w|=\sqrt{1-|q|^{2}}$. In the third case the optimality condition (A.3) reads

[TABLE]

We multiply the optimality condition by $q$ and obtain

[TABLE]

Multiplying (A.5) by $w$ yields

[TABLE]

and another multiplication of (A.5) by $f_{\eta}$ yields

[TABLE]

where we inserted the previous results in the last two steps. We rearrange and find

[TABLE]

which finally leads us to

[TABLE]

Hence we have

[TABLE]

It remains to show that

[TABLE]

At first we observe that for any $g\in L_{|\eta|}^{1}(\Omega,\mathbb{R}^{m})$ we have that $z_{*}:=g|\eta|\in\mathcal{M}(\Omega,\mathbb{R}^{m})$, since

[TABLE]

By the definition of the dual norm we furthermore obtain

[TABLE]

For $|\eta|$-a.e. $x\in\Omega$ we define

[TABLE]

where we again leave out the dependence on $x$ in the following. We have to distinguish four cases.

1st case: If $|q|<\cos(\varphi)$, we have $q\cdot f_{\eta}>0$ and we can obtain

[TABLE]

for $g=0$.

2nd case: Analogously if $|q|<-\cos(\varphi)$, we have $q\cdot f_{\eta}<0$ and $g=f_{\eta}$ leads to

[TABLE]

3rd case: In case $|q|=1$ we let $c>0$ and $g=f_{\eta}/2-(cq)/2$ to find

[TABLE]

Using a Taylor expansion around $q$ and the fact that $|q|=1$ we obtain

[TABLE]

which leads us to

[TABLE]

for $c\to\infty$. Hence for every $\varepsilon>0$ there exists a $c^{*}>0$ such that $H(g)\leq\varepsilon/\|\eta\|_{\mathcal{M}}$.

4th case: Finally, if $|q|<1$ and $|q|\geq|\cos(\varphi)|$, we can pick $g=2\lambda_{1}(w-q)$ with $\lambda_{1}$ and $w$ being the Lagrange multiplier and dual variable from the above computations of the third case. We recall that $|w+q|=|w-q|=1$, $w\cdot q=0$, $|w|=\sqrt{1-|q|^{2}}$ and $f_{\eta}-2\lambda_{1}(w-q)=2\lambda_{2}(w+q)$. A straight forward calculation yields

[TABLE]

Hence for $|\eta|$-a.e. $x\in\Omega$ we define $g\colon\Omega\to\mathbb{R}^{m}$ as

[TABLE]

For every $\alpha>0$ we let

[TABLE]

$g^{\alpha}$ is bounded $|\eta|$-a.e., thus $g^{\alpha}\in L_{|\eta|}^{1}(\Omega,\mathbb{R}^{m})$ and $z_{*}^{\alpha}:=g^{\alpha}|\eta|\in\mathcal{M}(\Omega,\mathbb{R}^{m})$. We remark that since $G$ is bounded by $1,$ $|\eta|$-a.e., we have $\int_{\Omega}G(f_{\eta},q)\,\mathrm{d}\eta\leq\|\eta\|_{\mathcal{M}}$, and compute

[TABLE]

for $\alpha\to\infty$. Since $\varepsilon$ was arbitrary, this completes the proof. ∎

A.2 Numerical solution of PET-MRI joint reconstruction

For the solution of the joint PET-MRI reconstruction scheme (4.4), (4.5) we only need to further specify the convex conjugates of the involved data terms and the role of the sets $C_{i}$. We use the notation from Section 4 in order to be consistent. Since we constrain both the PET and the MR image to be real-valued and positive, we let $C=\mathbb{R}_{0}^{+}$. Hence the full problem reads:

[TABLE]

Since both data terms are differentiable, their convex conjugates are easy to compute. The convex conjugate of the scaled Kullback-Leibler divergence

[TABLE]

is given by

[TABLE]

The associated proximal map is

[TABLE]

For the quadratic MR data term

[TABLE]

we derive the conjugate

[TABLE]

and the proximal map

[TABLE]

The primal proximal map for $\delta_{C}$ can be carried out as a simple projection onto the nonnegative real numbers. Regarding step sizes, we employ the diagonal preconditioning presented in [43], using matrix representations of our operators to precompute $\sigma$ and $\tau$.

Acknowledgements

This work has been supported by ERC via Grant EU FP 7 - ERC Consolidator Grant 615216 LifeInverse. MB acknowledges support by the German Science Foundation DFG via SFB 656 Molecular Cardiovascular Imaging, Subproject B2, and EXC 1003 Cells in Motion Cluster of Excellence, Münster, Germany. The authors thank Frank Wübbeling (WWU), Florian Knoll (NYU), and Matthias Erhardt (Cambridge) for helpful and stimulating discussions. Some additional thanks go to the latter for providing easily usable code alongside his work and figuring out the nicest way to design figures for joint reconstruction papers.