Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation

TL;DR

This paper introduces a novel end-to-end training strategy for CNNs that incorporates anatomical prior knowledge through a regularisation model, improving cardiac image analysis and enabling shape-based pathology classification.

Contribution

It presents a generic framework that embeds anatomical priors into CNN training, enhancing segmentation and image enhancement tasks with shape-aware regularisation.

Findings

Improved accuracy in cardiac image segmentation and enhancement.

Framework adaptable to various analysis tasks.

Deep shape models serve as biomarkers for cardiac pathologies.

Abstract

Incorporation of prior knowledge about organ shape and location is key to improve performance of image analysis approaches. In particular, priors can be useful in cases where images are corrupted and contain artefacts due to limitations in image acquisition. The highly constrained nature of anatomical objects can be well captured with learning based techniques. However, in most recent and promising techniques such as CNN based segmentation it is not obvious how to incorporate such prior knowledge. State-of-the-art methods operate as pixel-wise classifiers where the training objectives do not incorporate the structure and inter-dependencies of the output. To overcome this limitation, we propose a generic training strategy that incorporates anatomical prior knowledge into CNNs through a new regularisation model, which is trained end-to-end. The new framework encourages models to follow the global anatomical properties of the underlying anatomy (e.g. shape, label structure) via learned non-linear representations of the shape. We show that the proposed approach can be easily adapted to different analysis tasks (e.g. image enhancement, segmentation) and improve the prediction accuracy of the state-of-the-art models. The applicability of our approach is shown on multi-modal cardiac datasets and public benchmarks. Additionally, we demonstrate how the learned deep models of 3D shapes can be interpreted and used as biomarkers for classification of cardiac pathologies.

Tables

Table 1. TABLE I: Stacks of 2D cardiac MR images ( 200 200 200 ) are segmented into LV endocardium and myocardium, and the segmentation accuracy is evaluated in terms of Dice metric and surface to surface distances. The ground-truth labels are obtained from high resolution 3D images acquired from same subjects, which do not contain motion and blocky artefacts. The proposed approach (ACNN-Seg) is compared against state-of-the-art slice by slice segmentation (2D-FCN [ 44 ] ) method, 3D-UNet model [ 12 ] , cascaded 3D-UNet and convolutional AE model (AE-Seg) [ 37 ] , proposed sub-pixel segmentation model (3D-Seg) and the same model with motion augmentation used in training (3D-Seg-MAug).

	Endocardium			Myocardium			Capacity
	Mean	Hausdorff	Dice	Mean	Hausdorff	Dice	# Trainable
	Dist. (mm)	Dist. (mm)	Score (%)	Dist. (mm)	Dist. (mm)	Score (%)	Parameters
2D-FCN [44]	2.07$\pm$0.61	11.37$\pm$7.15	.908$\pm$.021	1.58$\pm$0.44	9.19$\pm$7.22	.727$\pm$.046	$1.39\times {10}^6$
3D-Seg	1.77$\pm$0.84	10.28$\pm$8.25	.923$\pm$.019	1.48$\pm$0.51	10.15$\pm$10.58	.773$\pm$.038	$1.60\times {10}^6$
3D-UNet [12]	1.66$\pm$0.74	9.94$\pm$9.22	.923$\pm$.019	1.45$\pm$0.47	9.81$\pm$11.77	.764$\pm$.045	$1.64\times {10}^6$
AE-Seg [37]	1.75$\pm$0.58	8.42$\pm$3.64	.926$\pm$.019	1.51$\pm$0.29	8.52$\pm$2.72	.779$\pm$.033	$1.68\times {10}^6$
3D-Seg-MAug	1.59$\pm$0.74	8.52$\pm$8.13	.928$\pm$.019	1.37$\pm$0.41	9.41$\pm$9.17	.785$\pm$.041	$1.60\times {10}^6$
AE-Seg-M	1.59$\pm$0.48	7.52$\pm$3.78	.927$\pm$.017	1.32$\pm$0.26	7.12$\pm$2.79	.791$\pm$.036	$1.91\times {10}^6$
ACNN-Seg	1.37$\pm$0.42	7.89$\pm$3.83	.939$\pm$.017	1.14$\pm$0.22	7.31$\pm$3.59	.811$\pm$.027	$1.60\times {10}^6$
p-values	$p\ll 0.001$	$p\approx 0.890$	$p\ll 0.001$	$p\ll 0.001$	$p\approx 0.071$	$p\ll 0.001$	-

Table 2. TABLE II: 3D-US cardiac image sequences (in total 30 30 30 ) are segmented into LV cavity and background. Segmentation accuracy is evaluated in terms of Dice score (DSC), surface-to-surface distances. The consistency of delineations on both ED and ES phases are measured in terms computed ejection fraction (EF) values. The proposed ACNN-Seg method is compared against state-of-the art deformable shape fitting [ 6 ] and fully-convolutional 3D segmentation [ 11 ] methods.

	End Diastole (ED)			End Systole (ES)
	BEAS [6]	FCN [11]	ACNN-Seg	BEAS [6]	FCN [11]	ACNN-Seg	p-values
Mean Dist (mm)	2.26$\pm$0.73	1.98$\pm$1.03	1.89$\pm$0.51	2.43$\pm$0.91	2.83$\pm$1.89	2.09$\pm$0.77
HD Dist (mm)	8.10$\pm$2.66	11.94$\pm$9.46	6.96$\pm$1.75	8.13$\pm$3.08	12.45$\pm$10.69	7.75$\pm$2.65
DSC (%)	.894$\pm$.041	.906$\pm$.026	.912$\pm$.023	.856$\pm$.057	.872$\pm$.050	.873$\pm$.051	$p\approx 0.05$
EF (Corr)	0.889	0.885	0.913	-	-	-	-
EF (Bias+LOA) (ml)	-6.78$\pm$27.71	2.74$\pm$12.01	1.78$\pm$10.09	-	-	-	-

Table 3. TABLE III: Average inference time (Inf-T) of the SR models per input LR image (120x120x12) using a GPU (GTX-1080). ACNN-SR and SR-CNN [ 34 ] models are given the same number of filters and capacity. MOS [ 26 ] results, received from the clinicians (R1 and R2), are reported separately.

	SSIM [47]	MOS-R1	MOS-R2	Inf-T
Linear	.777$\pm$.043	2.71$\pm$0.82	2.60$\pm$.91	-
B-Spline	.779$\pm$.053	2.77$\pm$0.89	2.64$\pm$.84	-
SR-CNN [34]	.783$\pm$.046	3.59$\pm$1.05	3.85$\pm$.70	.29 s
3D-UNet [12]	.784$\pm$.045	3.55$\pm$0.92	3.99$\pm$.71	.07 s
ACNN-SR	.796$\pm$.041	4.36$\pm$0.62	4.25$\pm$.68	.06 s
p-values	$p\ll 0.001$			-

Table 4. TABLE IV: Structure of the predictor model: The model maps the input HR intensity image ( 120 120 120 x 120 120 120 x 60 60 60 ) to the latent space and generates a 64 64 64 -dim representation. The size, number, and stride of the learnt convolution (Conv) kernels are provided. The filters operate on different image scales ( S ​ 1 𝑆 1 S1 - S ​ 4 𝑆 4 S4 ) and each convolution operation is followed by a non-linear unit (ReLU).

		Size	Stride	# Kernels	Non-linearity
$S1$	Conv	(f:3,3,3)	(s:1,1,1)	(N:32)	ReLU
	Conv	(f:3,3,3)	(s:1,1,1)	(N:32)	ReLU
$S2$	Conv	(f:3,3,3)	(s:2,2,2)	(N:64)	ReLU
	Conv	(f:3,3,3)	(s:1,1,1)	(N:64)	ReLU
$S3$	Conv	(f:3,3,3)	(s:2,2,2)	(N:128)	ReLU
	Conv	(f:3,3,3)	(s:1,1,1)	(N:128)	ReLU
$S4$	Conv	(f:3,3,3)	(s:2,2,2)	(N:256)	ReLU
	Conv	(f:3,3,3)	(s:1,1,1)	(N:1)	ReLU
	FC	-	-	(N:64)	None

Table 5. TABLE V: Structure of the autoencoder (AE) model: The encoder part maps the given input segmentation map ( 120 120 120 x 120 120 120 x 60 60 60 ) to the latent space through convolution (Conv) and fully-connected (FC) layers. The decoder part recovers the input from the low-dimensional representation and outputs a segmentation map ( 120 120 120 x 120 120 120 x 60 60 60 ). The size, number, and stride of the learnt convolution (Conv) kernels are provided. The filters operate on different image scales ( S ​ 1 𝑆 1 S1 - S ​ 4 𝑆 4 S4 ) and each convolution operation is followed by a non-linear unit (ReLU).

		Kernel	Stride	# Kernels	NonLin
$S1$	Conv	(f:3,3,3)	(s:2,2,1)	(N:16)	ReLU
	Conv	(f:3,3,3)	(s:1,1,1)	(N:16)	ReLU
$S2$	Conv	(f:3,3,3)	(s:2,2,2)	(N:32)	ReLU
	Conv	(f:3,3,3)	(s:1,1,1)	(N:32)	ReLU
$S3$	Conv	(f:3,3,3)	(s:2,2,2)	(N:64)	ReLU
	Conv	(f:3,3,3)	(s:1,1,1)	(N:64)	ReLU
$S4$	Conv	(f:3,3,3)	(s:3,3,3)	(N:1)	ReLU
$HC$	FC	-	-	(N:64)	None
	FC	-	-	(N:125)	ReLU
$S4$	Deconv	(f:7,7,7)	(s:3,3,3)	(N:64)	ReLU
	Conv	(f:3,3,3)	(s:1,1,1)	(N:64)	ReLU
$S3$	Deconv	(f:4,4,4)	(s:2,2,2)	(N:32)	ReLU
	Conv	(f:3,3,3)	(s:1,1,1)	(N:32)	ReLU
$S2$	Deconv	(f:4,4,4)	(s:2,2,2)	(N:16)	ReLU
	Conv	(f:3,3,3)	(s:1,1,1)	(N:16)	ReLU
$S1$	Deconv	(f:4,4,1)	(s:2,2,1)	(N:16)	ReLU
	Conv	(f:3,3,3)	(s:1,1,1)	(N:3)	None

Equations

$$\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\begin{gathered}L_{h_{e}}=\left\lVert\,f(\phi(\boldsymbol{x});\boldsymbol{\theta}_{f})-f(\boldsymbol{y};\boldsymbol{\theta}_{f})\,\right\rVert_{2}^{2}\\[2.84526pt] \min\limits_{\boldsymbol{\theta}_{s}}\>\left(L_{x}\,(\phi(\boldsymbol{x};\boldsymbol{\theta}_{s}),\boldsymbol{y})+\lambda_{1}\cdot L_{h_{e}}+\frac{\lambda_{2}}{2}||\boldsymbol{w}||_{2}^{2}\right)\end{gathered}$$

$$\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\begin{gathered}L_{h_{p}}=\left\lVert\,p\,(\Phi(\boldsymbol{x});\boldsymbol{\theta}_{p})-p\,(\boldsymbol{y}_{r};\boldsymbol{\theta}_{p})\,\right\rVert_{2}^{2}\\[2.84526pt] \min\limits_{\boldsymbol{\theta}_{r}}\,\left(\Psi_{\ell_{1}}\left(\Phi(\boldsymbol{x}\,;\boldsymbol{\theta}_{r})-\boldsymbol{y}_{r}\right)+\lambda_{1}\cdot L_{h_{p}}+\frac{\lambda_{2}}{2}||\boldsymbol{w}||_{2}^{2}\right)\end{gathered}$$

Full text

I Introduction

Image segmentation techniques aim to partition an image into meaningful parts which are used for further analysis. The segmentation process is typically driven by both the underlying data and a prior on the solution space, where the latter is useful in cases where the images are corrupted or contain artefacts due to limitations in the image acquisition. For example, bias fields, shadowing, signal drop-out, respiratory motion, and low-resolution acquisitions are the few common limitations in ultrasound (US) and magnetic resonance (MR) imaging.

Incorporating prior knowledge into image segmentation algorithms has proven useful in order to obtain more accurate and plausible results as summarised in the recent survey [32]. Prior information can take many forms: boundaries and edge polarity [10]; shape models [13, 14]; topology specification; distance prior between regions; atlas models [5], which were commonly used as a regularisation term in energy optimisation based traditional segmentation methods (e.g. region growing). In particular, atlas priors are well suited for medical imaging applications since they enforce both location and shape priors through a set of annotated anatomical atlases. Similarly, auto-context models [45] have made use of label and image priors in segmentation, which require a cascade of models.

In the context of neural networks (NNs), early work on shape analysis has focused on learning generative models through deep Boltzmann Machines (DBMs), namely ShapeBM [18] that uses a form of DBM with sparse pixel connectivity. Follow-up work in [9, 17] has demonstrated the application of DBMs to binary segmentation problems in natural images containing vehicles and other types of objects. However, fully connected DBM for images require a large number of parameters and consequently model training may become intractable depending on the size of images. For this reason, convolutional deep belief nets [48] were recently proposed for encoding shape prior information. Besides variational models, cascaded convolutional architectures [27, 37] have been shown to discover priors on shape and structure in label space without any a priori specification. However, this comes at the cost of increased model complexity and computational needs.

In the context of medical imaging and neural networks, anatomical priors have not been studied in much depth, particularly in the current state-of-the-art segmentation techniques [22, 38, 11, 36]. Recent work has shown simple use cases of priors through adjacency [7] and boundary [10] conditions. Inclusion of priors in medical imaging could potentially have much more impact compared to their use in natural image analysis since anatomical objects in medical images are naturally more constrained in terms of their shape and location.

As explained in a recent NN survey paper [28], the majority of the classification and regression models utilise a pixel-level loss function (e.g. cross-entropy or mean square error) which does not fully take into account the underlying semantic information and dependencies in the output space (e.g. class labels). In this paper, we present a novel and generic way to incorporate global shape/label information into NNs. The proposed approach, namely anatomically constrained neural networks (ACNN), is mainly motivated by the early work on shape priors and image segmentation, in particular PCA based statistical [13] and active shape models [14]. Our framework learns a non-linear compact representation of the underlying anatomy through a stacked convolutional autoencoder [31] and enforces network predictions to follow the learnt statistical shape/label distributions. In other words, it favours predictions that lie on the extracted low dimensional data manifold. More importantly, our approach is independent of the particular NN architecture or application; it can be combined with any of the state-of-the-art segmentation or super-resolution (SR) NN models and potentially improve its prediction accuracy and robustness without introducing any memory or computational complexity at inference time. Lastly, ACNN models, trained with the proposed prior term which acts as a regulariser, remove the need for post-processing steps such as conditional random fields [24] which are often based on heuristics parameter tuning. In ACNN, the regularisation is part of the end-to-end learning which can be a great advantage.

The proposed global training objective in SR corresponds to a prior on the space of feasible high-resolution (HR) solutions, which is experimentally shown to be useful since SR is an ill-posed problem. Similar modifications of the objective function during training have been introduced to enhance the quality of natural images, such as perceptual [21] and adversarial [26] loss terms, which were used to synthesise more realistic images in terms of texture and object boundaries. In the context of medical imaging, our priors enforce the synthesised HR images to be anatomically meaningful while minimising a traditional image reconstruction loss function.

I-A Clinical Motivation

Cardiac imaging has an important role in diagnosis, pre-operative planning, and post-operative management of patients with heart disease. Imaging modalities such as US and cardiac MR (CMR) are widely used to provide detailed assessment of cardiac function and morphology. Each modality is suitable for particular clinical use cases; for instance, 2D-US is still the first line of choice due to its low cost and wide availability, whereas, CMR is a more comprehensive modality with excellent contrast for both anatomical and functional evaluation of the heart [23]. Similarly, 3D-US is recommended over the use of 2D-US since it has been demonstrated to provide more accurate and reproducible volumetric measurements [25].

Some of the standard clinical acquisition protocols in 3D-US and CMR still have limitations in visualising the underlying anatomy due to imaging artefacts (e.g. cardiac motion, low slice resolution, lack of slice coverage [35]) or operator-dependent errors (e.g. shadows, signal drop-outs). In the clinical routine, these challenges are usually tackled through multiple acquisitions of the same anatomy and repeated patient breath-holds leading to long examination times. Similar problems have been reported in large cohort studies such as the UK Biobank [35], which leads to inaccurate quantitative measurements or even the discarding of acquired images. As can be seen in Fig. 1, the existing state-of-the-art convolutional neural network (CNN) approaches for segmentation [44, 11] and image enhancement [34] tasks perform poorly when the input data is not self-consistent for the analysis. For this reason, incorporation of prior knowledge into cardiac image analysis could provide more accurate and reliable assessment of the anatomy, which is shown in the third column of the same figure. Most importantly, the proposed ACNN model allows us to perform HR analysis via sub-pixel feature maps generated from low resolution (LR) input data even in the presence of motion artefacts. Using the proposed approach we can perform full 3D segmentation without explicit motion correction and do not have to rely on LR slice-by-slice 2D segmentation.

We demonstrate the applicability of the proposed approach for cine stacks of 2D MR and 3D-US datasets composed of $1200$ and $45$ cardiac image sequences respectively. We show that the proposed segmentation and SR models become more robust against imaging artefacts mentioned earlier which is underlined by our state-of-the-art results on the MICCAI’14 CETUS public benchmark [8]. We also demonstrate that the lower dimensional representations learnt by the proposed ACNN can be useful for classification of pathologies such as dilated and hypertrophic cardiomyopathy, and it does not require point-wise correspondence search between subjects as in [39]. For the evaluation, the MICCAI’17 AC/DC classification benchmark was used. In that regard, the proposed method is not only useful for image enhancement and segmentation but also for the study of anatomical shape variations in population studies and their associations with cardiac related pathologies.

I-B Contributions

In this study, we propose a generic and novel technique to incorporate priors on shape and label structure into NNs for medical image analysis tasks. In this way, we can constrain the NN training process and guide the NN to make anatomically more meaningful predictions, in particular in cases where the input image data is not informative or consistent enough (e.g. missing object boundaries). More importantly, to the best of our knowledge, this is one of the earliest studies demonstrating the use of convolutional autoencoder networks to learn anatomical shape variations from medical images.

The proposed ACNN model is evaluated on multi-modal cardiac datasets from MR and US. Our evaluation shows: (I) A sub-pixel cardiac MR image segmentation approach that, in contrast to previous CNN approaches [44, 2], is robust against slice misalignment and coverage problems; (II) An implicit statistical parametrisation of the left ventricular shape via NNs for pathology classification; (III) An image SR technique that extends previous work [34] and that is robust against slice misalignments; our approach is computationally more efficient than the state-of-the-art SR-CNN model [34] as the feature extraction is performed in the low-dimensional image space. (IV) Last, we demonstrate state-of-the-art 3D-US cardiac segmentation results on the CETUS’14 Benchmark.

II Methodology

In the next section, we briefly summarise the state-of-the-art methodology for image segmentation (SEG) and super-resolution (SR), which is based on convolutional neural networks (CNNs). We then present a novel methodology that extends these CNN models with a global training objective to constrain the output space by imposing anatomical shape priors. For this, we propose a new regularisation network that is based on the T-L architecture which was used in computer graphics [19] to $3$D render objects from natural images.

II-A Medical Image Segmentation with CNN Models

Let $\boldsymbol{y}_{s}=\{y_{i}\}_{i\in\mathcal{S}}$ be an image of class labels representing different tissue types with $y_{i}\in\mathcal{L}=\{1,2,\ldots C\}$. Furthermore let $\boldsymbol{x}=\{x_{i}\in\mathbb{R},i\in\mathcal{S}\}$ be the observed intensity image. The aim of image segmentation is to estimate $\boldsymbol{y}_{s}$ having observed $\boldsymbol{x}$. In CNN based segmentation models [22, 29, 38], this task is performed by learning a discriminative function that models the underlying conditional probability distribution $P(\boldsymbol{y}_{s}|\boldsymbol{x})$.

The estimation of class densities $P(\boldsymbol{y}_{s}|\boldsymbol{x})$ consists in assigning to each $x_{i}$ the probability of belonging to each of the $C$ classes, yielding $C$ sets of class feature maps $f_{c}$ that are extracted through learnt non-linear functions. The final decision for class labels is then made by applying softmax to the extracted class feature maps, in the case of cross-entropy $L_{x}=-\sum_{c=1}^{C}\,\sum_{i\in\mathcal{S}}\,\log\left(\frac{e^{f_{(c,i)}}}{\sum_{j}e^{f_{(j,i)}}}\right)$ these feature maps correspond to log likelihood values.

As in the U-Net [38] and DeepMedic [22] models, we learn the mapping between intensities and labels $\phi(\boldsymbol{x}):\mathcal{X}\rightarrow\mathcal{L}$ by optimising the average cross-entropy loss of each class $L_{x}=\sum_{c=1}^{C}\,L_{(x,c)}$ using stochastic gradient descent. As shown in Fig. 2, the mapping function $\phi$ is computed by passing the input image through a series of convolution layers and rectified linear units across different image scales to enlarge the model’s receptive field. The presented model is composed of two parts: feature extraction (analysis) similar to a VGG-Net [42] and reconstruction (synthesis) as in the case of a 3D U-Net [38]. However, in contrast to existing approaches, we aim for sub-pixel segmentation accuracy by training up-sampling layers with high-resolution ground-truth maps. This enables $3$D analysis of the underlying anatomy in case of thick slice 2D image stack acquisitions such as cine cardiac MR imaging. In this way, it is possible to perform analysis on the high-resolution image grid without any preceding upsampling operation with a SR model [34].

Similar segmentation frameworks (cf. [28]) have been studied in medical imaging. However, in most of the existing methods, the models are supervised purely through a local loss function at pixel level (e.g. cross-entropy, Dice) without exploiting the global dependencies and structure in the output space. For this reason, the global description of predictions is usually not adhering to shape, label, or atlas priors. In contrast to this we propose a model that can incorporate the aforementioned priors in segmentation models. The proposed framework relies on autoencoder and T-L network models to obtain a non-linear compact representation of the underlying anatomy, which are used as priors in segmentation.

II-B Convolutional Autoencoder Model and ACNN-Seg

An autoencoder (AE) [46] is a neural network that aims to learn an intermediate representation from which the original input can be reconstructed. Internally, it has a hidden layer $\boldsymbol{h}$ whose activations represent the input image, often referred as codes. To avoid the AE to directly copy its output, the AE are often designed to be undercomplete so that the size of the code is less than the input dimension as shown in Fig. 3. Learning an AE forces the network to capture the most salient features of the training data. The learning procedure minimises a loss function $L_{x}(\boldsymbol{y}_{s},g(f(\boldsymbol{y}_{s})))$, where $L_{x}$ is penalising $g(f(\boldsymbol{y}_{s}))$ being dissimilar from $\boldsymbol{y}_{s}$. The functions $g$ and $f$ are defined as the decoder and encoder components of the AE.

In the proposed method, the AE is integrated into the standard segmentation network, described in Sec. II-A, as a regularisation model to constrain class label predictions $\boldsymbol{y}$ towards anatomically meaningful and accurate outputs. The cross-entropy loss function operates on individual pixel level class predictions, which does not guarantee global consistency and plausible anatomical shapes even though the segmentation network has a receptive field larger than the size of structures to be segmented. This is due to the fact that back-propagated gradients are parametrised only by pixel-wise individual probability divergence terms and thus provide little global context.

To overcome this limitation, class prediction label maps are passed through the AE to obtain a lower dimensional (e.g. $64$ dimensions) parametrisation of the segmentation and its underlying structure [40]. By performing AE-based non-linear lower dimensional projections on both predictions and ground-truth labels, as shown in Fig. 4, we can build our ACNN-Seg training objective function though a linear combination of cross-entropy ($L_{x}$), shape regularisation loss ($L_{h_{e}}$), and weight decay terms as follows:

[TABLE]

Here $\boldsymbol{w}$ corresponds to weights of the convolution filters, and $\boldsymbol{\theta_{s}}$ denotes all trainable parameters of the segmentation model and only these parameters are updated during training. The coupling parameters $\lambda_{1}$ and $\lambda_{2}$ determine the weights of shape regularisation loss and weight decay terms used in the training. In this equation, the second term $L_{h_{e}}$ ensures that the generated segmentations are in a similar low dimensional space (e.g. shape manifold) as the ground-truth labels. In addition to imposing shape regularisation, this parametrisation encourages label consistency in model predictions, and reduces false-positive detections as they can influence the predicted codes in the hidden layer. The third term corresponds to weight decay to limit the number of free parameters in the model to avoid over-fitting. The proposed AE model is composed of convolutional layers and a fully connected layer in the middle as shown in Fig. 3, which is similar to the stacked convolutional autoencoder model proposed in [31]. The AE model details (e.g. layer configuration, parameter choices) are provided in the supplementary material.

II-C Medical Image Super-Resolution (SR) with CNNs

Super-resolution (SR) image generation is an inverse problem where the goal is to recover spatial frequency information that is outside the spatial bandwidth of the low resolution (LR) observation $\boldsymbol{x}\in\mathbb{R}^{N}$ to predict a high resolution (HR) image $\boldsymbol{y}_{r}\in\mathbb{R}^{M}$ ($N\ll M$), as illustrated in the top row of Fig. 1. Since the high frequency components are missing in the observation space, usually training examples are used to predict the most likely $P(\boldsymbol{y}_{r}|\boldsymbol{x})$ HR output. Image SR is an ill-posed problem as there are an infinite number of solutions for a given input sample but only a few would be anatomically meaningful and accurate. As for the case of image segmentation, learnt shape representations can be used to regularise image SR, constraining the model to make only anatomically meaningful predictions.

Similar to the SR framework described in [34], our proposed SR model learns a mapping function $\Phi:\mathcal{X}\rightarrow\mathcal{Y}$ to estimate a high-resolution image $\hat{\boldsymbol{y}}_{r}=\Phi(\boldsymbol{x}\,;\boldsymbol{\theta}_{r})$ where $\boldsymbol{\theta}_{r}$ denotes the model parameters such as convolution kernels and batch-normalisation statistics. The parameters are optimised by minimising the smooth $\ell_{1}$ loss, also known as Huber loss, between the ground-truth high resolution image and the corresponding prediction. The smooth $\ell_{1}$ norm is defined as $\Psi_{\ell_{1}}(k)=\{0.5\,k^{2}\;\text{if}\;|k|<1\,,\,|k|-0.5\;\text{otherwise}\}$ and the SR training objective becomes $\min\limits_{\boldsymbol{\theta}_{r}}\,\sum_{i\in\mathcal{S}}\,\Psi_{\ell_{1}}\left(\Phi(\boldsymbol{x}_{i}\,;\boldsymbol{\theta}_{r})-\boldsymbol{y}_{i}\right)$

In the proposed SR framework, we used the same model as shown in Fig. 2. It provides two main advantages over the state-of-the-art medical image SR model proposed in [34]: (I) the network generates image features in the LR image grid rather than early upsampling of the features, which reduces memory and computation requirements significantly. As highlighted in [41], early upsampling introduces redundant computations in the HR space since no additional information is added into the model by performing transposed convolutions [49] at an early stage. (II) The second advantage is the use of a larger receptive field to learn the underlying anatomy, which was not the case in earlier SR methods used in medical imaging [34] and natural image analysis [41, 16] because these models usually operate on local patch level. Capturing large context indeed helps our model to better understand the underlying anatomy and this enables us to enforce global shape constraints. This is achieved by generating SR feature-maps in multiple scales using multi strides in the in-plane direction.

Similar to the ACNN-Seg model, it is possible to regularise SR models to synthesise anatomically more meaningful HR images. To achieve this goal, we extend the standard AE model to the T-L model which enables us to obtain shape representation codes directly from the intensity space. The idea is motivated by the recent work [19] on $3$D shape analysis in natural images. In the next section we will explain the training strategy and the use of the T-L model as a regulariser.

II-D T-L Network Model and SR-ACNN

Shape encoding AE models operate only on the segmentation masks and this limits its application to SR problem where the model output is an intensity image. To circumvent this problem, we extend the standard denoising AE to the T-L regularisation model by combining the AE with a predictor network (Fig. 3) $p(\boldsymbol{x}):\mathcal{X}\rightarrow\mathcal{H}$. The predictor can map an input image into a low dimensional non-parametric representation of the underlying anatomy (e.g. shape and class label information), which is learnt by the AE. In other words, it enables us to learn a hidden representation space that can be reached by non-linear mappings from both image label space $\mathcal{Y}$ and image intensity space $\mathcal{X}$. In this way, SR models can be regularised as well with respect to learnt anatomical priors.

This network architecture is useful in image analysis applications for two main reasons: (I) It enables us to build a regularisation network that could be used in applications different than image segmentation such as image SR. We propose to use this new regularisation network at training time of SR to enforce the models to learn global information about the images besides the standard pixel-wise ($\ell_{1}$ distance) image reconstruction loss. In this way, the regressor SR model is guided by the additional segmentation information, and it becomes robust against imaging artefacts and missing information. (II) The second important feature of the T-L model is the generalisation of the learnt representations. Joint training of the AE and predictor enables us to learn representations that could be extracted from both intensity and label space. The learnt codes will encode the variations that could be interpreted from both manual annotations and intensity images. Since a perfect mapping between the intensity and label spaces is practically not achievable, the T-L learnt codes are expected to be more representative due to the inclusion of additional information.

The T-L model is trained in two stages: In the first stage, the AE is trained separately with ground-truth segmentation masks and cross-entropy loss $L_{x}$. Later, the predictor model is trained to match the learnt latent space $\boldsymbol{h}$ by minimising the Euclidean distance $L_{h}$ between the codes predicted by the AE and predictor as shown in Fig. 3. Once the loss functions for both the AE and the predictor converge, the two models are trained jointly in the second stage. The encoder $f$ is updated using two separate back-propagated gradients $(\frac{\partial L_{x}}{\partial\theta_{f}}\,,\frac{\partial L_{h}}{\partial\theta_{f}})$ and the two loss functions are scaled to match their range. The first gradient encourages the encoder to generate codes that could be easily extracted by the predictor while the second gradient making sure that a good segmentation-reconstruction can be obtained at the output of the decoder. Training details are further discussed in Section III-B. It is important to note that the T-L regulariser model is used only at training time but not during inference; in other words, the fully convolutional (FCN) segmentation and super-resolution models can still be used for applications using different image sizes. In this paper, the proposed SR model is referred to as ACNN-SR and its training scheme is shown in the bottom part of Fig. 4.

[TABLE]

The training objective shown above is composed of weight decay, pixel-wise and global loss terms. Here $\lambda_{1}$ and $\lambda_{2}$ determine the weight of shape priors and weight decay terms while the smooth $\ell_{1}$ norm loss function $\Psi$ quantifies the reconstruction error. The global loss $L_{h_{p}}$ is defined as the Euclidean distance between the codes generated from the synthesised and ground-truth HR images. The T-L model is used only in the network training phase as a regularisation term, similar to VGG features [42] that were used for representing a perceptual loss function [21]. However, we are not interested in expanding the output space to a larger feature-map space, but instead obtain a compact representation of the underlying anatomy.

II-E Learnt Hidden Representations

The learnt low dimensional representation $\boldsymbol{h}$ is used to constrain NN models. Low dimensional encoding enables us to train models with global characteristics but also yields better generalisation power for the underlying anatomy as shown in earlier work [43]. However, since we update our segmentation and SR model parameters with the gradients back-propagated from the global loss layer using the Euclidean distance of these representations, it is essential to analyse the distribution of the extracted codes. In Fig. 5, due to space limitations, we show the histogram of $16$ randomly chosen codes (out of $64$) of a T-L model trained with cardiac MR segmentations. Note that each histogram is constructed using the corresponding code for every sample in the full dataset. It is observed that the learnt latent representations in general follow a normal distribution and they are not separated in multi-clusters (e.g. mixture of Gaussians). A smooth distribution of the codes ensures better supervision for the main NN model (SR, Seg) since the global gradients are back-propagated by computing the Euclidean distance between the obtained distributions.

This observation can be explained by the fact that the proposed T-L network is trained with small Gaussian input noise as in the case of denoising autoencoders. In [1], Alain and Bengio showed that the denoising reconstruction error is equivalent to contractive penalty, which forces the feature extraction (encoder) function $f$ resist perturbations of the input and contracts these input samples to similar low dimensional codes. The penalty is defined as $\Omega(\boldsymbol{h})=\lambda\left\lVert\frac{\partial f(\boldsymbol{x})}{\partial\boldsymbol{x}}\right\rVert_{F}^{2}$, where $F$ denotes the Frobenius norm (sum of squared elements), and $\boldsymbol{h}=f(\boldsymbol{x})$ represents the codes. The given penalty function promotes the network to learn the underlying low-dimensional data manifold and capture its local smooth structure. In addition to the smoothness of the latent distributions, the extracted codes are expected to be correlated since the decoder merges some of the codes along the three spatial dimensions to construct input feature maps for the transposed convolutions, but this characteristic is not a limitation in our study.

III Applications and Experiments

In this section, we present three different applications of the proposed ACNN model: 3D-US and cardiac MR image segmentation, as well as cardiac MR image SR. The experiments focus on demonstrating the importance of shape and label priors for image analysis. Additionally, we analyse the salient information stored in the learnt hidden representations and correlate them with clinical indices, showing their potential use as biomarkers for pathology classification. The next subsection describes the clinical datasets used in our experiments.

III-A Clinical Datasets

III-A1 UK Digital Heart Project Dataset

This dataset 111https://digital-heart.org/ is composed of $1200$ pairs of cine 2D stack short-axis (SAX) and cine 3D high resolution (HR) cardiac MR images. Each image pair is acquired from a healthy subject using a standard imaging protocol [4, 15]. In more detail, the 2D stacks are acquired in different breath-holds and therefore may contain motion artefacts. Similarly, 3D imaging is not always feasible in the standard clinical setting due to the requirements for long image acquisition. The voxel resolution of the images are fixed to $1.25$x$1.25$x$10.00$ mm and $1.25$x$1.25$x$2.00$ mm for 2D stack low resolution (LR) and HR images respectively. Dense segmentation annotations for HR images are obtained by manually correcting initial segmentations generated with a semi-automatic multi-atlas segmentation method [5], and all the annotations are performed on the HR images to minimise errors introduced due to LR in through plane direction. Since the ground-truth information is obtained from the HR motion-free images, the experimental results are expected to reflect the performance of the method with respect to an appropriate reference. The annotations consist of pixel-wise labelling of endocardium and myocardium classes. Additionally, the residual spatial misalignment between the 2D LR stacks and HR volumes is corrected using a rigid transformation estimated by an intensity based image registration algorithm.

III-A2 CETUS’14 Challenge Dataset

CETUS’14 segmentation challenge [8] is a publicly available platform 222https://www.creatis.insa-lyon.fr/Challenge/CETUS/index.html to benchmark cardiac 3D ultrasound (US) left-ventricle (LV) segmentation methods. The challenge dataset is composed of 3D+time US image sequences acquired from $15$ healthy subjects and $30$ patients diagnosed with myocardial infarction or dilated cardiomyopathy. The images were acquired from apical windows and LV chamber was the main focus of analysis. Resolution of the images was fixed to $1$ mm isotropic voxel size through linear interpolation. The associated manual contours of the LV boundary were drawn by three different expert cardiologists, and the annotations were performed only on the frames corresponding to end-diastole (ED) and end-systole (ES) phases. Method evaluation is performed in a blinded fashion on the testing set ($30$ out of $45$) using the MIDAS web platform.

III-A3 ACDC MICCAI’17 Challenge Dataset

The aim of the ACDC’17 challenge 333https://www.creatis.insa-lyon.fr/Challenge/acdc/ is to compare the performance of automatic methods for the classification of MR image examinations in terms of healthy and pathological cases: infarction, dilated cardiomyopathy, and hypertrophic cardiomyopathy. The publicly available dataset consists of $20$ (per class) cine stacks of 2D MR image sequences which are annotated at ED and ES phases by a clinical expert. In the experiments, latent representations (codes) extracted with the proposed T-L network are used to classify these images.

III-B Training Details of the Proposed Model

In this section, we discuss the details of data augmentation used in training, and also the optimisation scheme of the T-L model training. To improve the model’s generalisation capability, the input training samples are artificially augmented using affine transformations, which is used in both the segmentation and T-L models. For the SR models, on the other hand, respiratory motion artefacts between the adjacent slices are simulated via in-plane rigid transformations that are defined for each slice independently. The corresponding ground-truth HR images are not spatially transformed; in this way, the models learn to output anatomically correct results when the input slices are motion corrupted. Additionally, additive Gaussian noise is applied to input intensity images to make the segmentation and super-resolution models more robust against image noise. For the AE, the tissue class labels are randomly swapped with the probability of $0.1$ to encourage the model to map slightly different segmentation masks to neighbouring points in the lower dimensional latent space. It ensures the smoothness of the learnt low-dimensional manifold space as explained in Section II-E.

In the joint training of the T-L network, parameters of the encoder model ($f$) are updated by the gradients originating from both the cross-entropy loss ($L_{x}$) and Euclidean distance terms ($L_{h}$). Instead of applying these two gradient descent updates sequentially in an iterative fashion, we perform a joint update training scheme and experimentally observed better convergence.

III-C Cardiac Cine-MR Image Segmentation

In this experiment, NN models are used to segment cardiac cine MR images in the dataset described in Sec. III-A1. As an input to the models, only the 2D stack LR images are used, which is a commonly used acquisition protocol for cardiac imaging, and the segmentation is performed only on the ED phase of the sequences. The corresponding ground-truth label maps, however, are projected from the HR image space, which are annotated in the HR image grid. The dataset ($1200$ LR images & HR labels) is randomly partitioned into three subsets: training ($900$), validation ($100$), and testing ($200$). All the images are linearly intensity normalised and cropped based on the automatically detected six anatomical landmark locations [33].

The proposed ACNN-Seg method is compared against: the current state-of-the-art cine MR 2D slice by slice segmentation method (2D-FCN) [44], 3D-UNet model [12], cascaded 3D-UNet and convolutional AE model (AE-Seg) [37], sub-pixel 3D-CNN segmentation model (3D-Seg) proposed in Sec. II-A, and the same model trained with various types of motion augmentation (3D-Seg-MAug). As the models have a different layout, the number of trainable parameters (pars) used in each model is kept fixed to avoid any bias. For the cascaded AE-Seg model, however, additional convolutional kernels are used in the AE as suggested in [37]. To observe the influence of the AE model’s capacity on the AE-Seg model’s performance, we performed experiments using different number of AE pars, and the largest capacity case is denoted by AE-Seg-M.

The results of the experiments are provided in Table II-E together with the capacity of each model. Statistical significance of the results is verified by performing the Wilcoxon signed-rank test between the top two performing methods for each evaluation metric. Based on these results we can draw three main conclusions: (I) Slice by slice analysis [2, 44] significantly under-performs compared to the proposed sub-pixel and ACNN-Seg segmentation methods. In particular, the dice score metrics are observed to be lower since 2D analysis can yield poor performance in basal and apical parts of the heart as shown in Fig. 6. Previous slice by slice segmentation approaches validated their methods on LR annotations; however, we see that the produced label maps are far off from the true underlying ventricular geometry and it can be a limiting factor for the analysis of ventricle morphology. Similar results were obtained in clinical studies [15], which however required HR image acquisition techniques. (II) The results also show that introduction of shape priors in segmentation models can be useful to tackle false-positive detections and motion-artefacts. As can be seen in the bottom row of Fig. 6, without the learnt shape priors, label map predictions are more prone to imaging artefacts. Indeed, it is the main reason why we observe such a large difference in terms of Hausdorff distance. For endocardium labels, on the other hand, the difference in dice score metric is observed to be less due to the larger size of the LV blood pool compared to the myocardium.

Lastly (III), we observe a performance difference between the cascaded AE based segmentation (AE-Seg [37]) and the proposed ACNN-Seg models: the segmentations generated with the former model are strongly regularised due to the second stage AE. It results in reduced Hausdorff distance with marginal statistical significance, but the model overlooks fine details of the myocardium surface since the segmentations are generated only from the coarse level feature-maps. More importantly, cascaded approaches add additional computational complexity due to the increased number of filters, which could be redundant given that the standard segmentation model is able to capture shape properties of the organs as long as it has a large receptive field and is optimised with shape constraints. In other words, shape constraints can be learnt and utilised in standard segmentation models, as shown in ACNN-Seg, without a need for additional model parameters and computational complexity. We also analysed the performance change in AE-Seg with respect to the number of parameters, which shows that the small capacity AE-Seg model ($8\times 10^{4}$ pars) is not suitable for cardiac image segmentation as the second stage in the cascaded model does not improve the performance significantly.

We performed additional segmentation experiments using only the T-L network. In detail, the input LR image is passed first through the predictor network and then the extracted codes are fed to the decoder network shown in Fig. 3. Label map predictions are collected at the output of the decoder and they are compared with the same ground-truth annotations described previously, which was similar to the AE based segmentation method proposed in [2, 3]. We observed that reconstruction of label-maps from low dimensional representations was limited since the ventricle boundaries were not delineated properly but rather a rough segmentation was generated (DSC: $.734$). We believe that this is probably the main reason why the authors of [2] proposed the use of a separate deformable model at the output of a NN. Nevertheless, the proposed ACNN-Seg does not require an additional post-processing step.

III-D Cardiac 3D Ultrasound Image Segmentation

In the second experiment, the proposed model is evaluated on 3D cardiac ultrasound data which is described in Sec III-A2. Segmentation models are used to delineate endocardial boundaries and the segmentations obtained on ED and ES frames are later used to measure volumetric indices such as ejection fraction (EF). The models are compared also in terms of surface to surface distance errors of their corresponding endocardium segmentations. As a baseline CNN method, we utilised the fully convolutional network model suggested by [11] for multi-view 3D-US image segmentation problem. It is also observed to be more memory efficient compared to the standard 3D-UNet architecture [12]. Additionally, we compare our proposed model against the CETUS’14 challenge winner approach (BEAS) [6] that utilised deformable models to segment the left ventricular cavity. The challenge results can be found in [8].

The experimental results, given in Table III-B, show that neural network models outperforms previous state-of-the-art approaches in this public benchmark dataset although the training data size was limited to 15 image sequences. The experimental results were evaluated in a blinded fashion by uploading the generated segmentations from separate 30 sequences into the CETUS web platform. The main contribution of ACNN model over the standard FCN approaches is the improved shape delineation of the LV, as it can be seen in terms of the distance error metrics. In particular, Hausdorff distances were reduced significantly as global regularisation reduces the amount of spurious false positive predictions and enforces abnormal LV segmentation shapes to fit into the learnt representation model. This situation is illustrated in Fig. 7. Similarly, we observed an improvement in terms of normalised Dice score, which was quantitatively not significant due to large volumetric size of the LV cavity. Lastly, we compared the extracted ejection fraction results to understand both the accuracy of segmentations and also the consistency of these predictions on both ED and ES phases. It is observed that the ACNN approach ensures better consistency between frames although none of the methods have used temporal information.

The reported results could be further improved by segmenting both ED and ES frames simultaneously or by extracting the temporal content from the sequences. For instance, propagation of ED masks to ES frames through optical flow has been shown to be a promising way to achieve this goal. However, this study mainly focuses on demonstrating the advantages of using priors in neural network models, and achieving the best possible segmentation accuracy was not our main focus.

III-E Cardiac MR Image Enhancement

The proposed ACNN model is also applied to the image SR problem and compared against the state-of-the-art CNN model used in medical imaging [34]. The cardiac MR dataset, described in Sec. III-A1, was split into two disjoint subsets: training ($1000$) and testing ($200$). At testing time, we evaluated our model with both LR-HR clinical image data. In training, however, LR images are synthetically generated from clinical HR data using the MR acquisition model discussed in [20]. More details about the acquisition model can be found in [34].

The quality of the upsampled images is evaluated in terms of SSIM metric [47] between the clinical HR image data and reconstructed HR images. SSIM measure assesses the correlation of local structures and is less sensitive to image noise than PSNR which is not used in our experiments since small misalignments between LR-HR image pairs could introduce large errors in the evaluation due to pixel by pixel comparisons. More importantly, intensity statistics of the images are observed to be different for this reason PSNR measurements would not be accurate. In addition to the SSIM metric, we used the mean opinion score (MOS) testing [26] to quantify the quality and similarity of the synthesised and real HR cardiac images. Two expert cardiologists were asked to rate the upsampled images from 1 (very poor) to 5 (excellent) based on the accuracy of the reconstructed LV boundary and geometry. To serve as a reference, the corresponding clinical LR and HR images are displayed together with the upsampled images that are anonymised for a fair comparison.

In Table III-E, SSIM and MOS scores for the standard interpolation techniques, SR-CNN, and the proposed ACNN-SR models are provided. In addition to the increased image quality, the ACNN-SR model is computationally more efficient in terms of run-time in comparison to the SR-CNN model [34] by a factor of $5$. This is due to the fact that ACNN-SR performs feature extraction in the low dimensional image space. Furthermore, we investigated the contribution of shape regularisation term in the application of SR, which is visualised in Fig. 8.

Moreover, we investigated the use of SR as a pre-processing technique for subsequent analysis such as image segmentation, similar to the experiments reported in [34]. In that regard, the proposed SR model and U-Net segmentation models are concatenated to obtain HR segmentation results. However, we observed that the proposed baseline sub-pixel segmentation model (3D-Seg), which merges both SR and segmentation tasks, performs better than the concatenated models. The 3D-Seg approach uses the convolution kernels more efficiently without requiring the model to output a high-dimensional intensity image. For this reason, SR models should be trained by taking into account the final goal and in some cases it’s not required to reconstruct a HR intensity image for HR analysis.

III-F Learnt Latent Representations and Pathology Classification

The jointly trained T-L model and its latent representations are analysed and evaluated in the experiment of image pathology classification. This experiment focuses on understanding the information stored in the latent space and also investigates whether they can be used to distinguish healthy subjects from dilated and hypertrophic cardiomyopathy patients. For this, we collected $64$ dimensional codes from segmentation images of the cardiac MR dataset explained in Sec. III-A3. Similarly, principal component analysis (PCA) was applied to the same segmentation images (containing LV blood-pool and myocardium labels) to generate $64$ dimensional linear projection of the labels, which requires additional spatial-normalisation prior to linear mapping. The generated codes were then used as features to train an ensemble of decision trees to categorise each image. We used $10$-fold cross-validation on $60$ CMR sequences and obtained $76.6\%$ vs $83.3\%$ accuracy using PCA and T-L codes extracted from ED phase. By including the codes from ES phase, the classification accuracies were improved to $86.6\%$ vs $91.6\%$. This result shows that although the AE and T-L models are not trained with the classification objective, they can still capture anatomical shape variations that are linked to cardiac related pathologies. In particular, we observed that some latent dimensions are more commonly used than others in tree node splits. By sampling codes from the latent space across these dimensions, we observed that the network captures the variation in wall thickness and blood pool size as shown in Fig. 9. Since we obtain a regular and smooth latent representation, it is possible to transverse along the latent space and generate LV shapes by interpolating between data points. It is important to note that classification accuracies can be further improved by training the AE and T-L models with a classification objective. Our main goal in this experiment was to understand whether the enforced prior distributions contain anatomical information or they are abstract representations only meaningful to the decoder of the AE.

IV Discussion and Conclusion

In this work, we presented a new image analysis framework that utilises autoencoder (AE) and T-L networks as regularisers to train neural network (NN) models. With this new training objective, at testing time NNs make predictions that are in agreement with the learnt shape models of the underlying anatomy, which are referred as image priors. The experimental results show that the state-of-the-art NN models can benefit from the learnt priors in cases where the images are corrupted and contain artefacts. The proposed regulariser model can be seen as an application-specific training objective. In that regard, our model differentiates from the VGG-Net [42] feature based training objectives [26, 21]. VGG features tend to be more general purpose representations that are learnt from ImageNet dataset containing natural images of a large variety of objects. In contrast to this, our AE model is trained solely on cardiac segmentation masks and features are customised to identify anatomical variations observed in the heart chambers. For this reason, we would expect the AE features of the segmentations to be more distinctive and informative.

As an alternative to the proposed framework, label space dependencies could be exploited also through adversarial loss (AL) objective functions. Such approaches have been used successfully in natural image super-resolution (SR) [26] and segmentation [30] tasks. In SR application, AL enables the SR network to hallucinate fine texture detail, and the synthesized HR images appear qualitatively more realistic. However, at the same time the PSNR and SSIM scores are usually worse. For this reason, the authors of [26] have pointed out that adversarial training may not be suitable for medical applications, where the accuracy and fidelity of the visual content more important than the qualitative appearance of the HR images. Moreover, we believe that adversarial training comes at the expense of less interpretability of the regularisation term and unstable model training behaviour, which still remains an open research problem.

Additionally, in the experiments we demonstrated that the learnt codes can be used as biomarkers for classification of cardiac related pathologies and we analysed the distribution of the learnt latent space. This latent space can be further constrained to be Gaussian distributed by replacing the proposed regularisation model with a variational autoencoder. However, this design choice was not considered in our ACNN framework due to two main reasons: (I) the additional K-L divergence term (constraint) would reduce the representation power of the AE; thus, the local anatomical variations would not be captured in detail. (II) A generative AE model is not essential for the regularisation of the proposed segmentation and SR models. A variational architecture would be useful if it was required to sample random instances from the latent space and reconstruct anatomically meaningful segmentation masks; however, in our framework we are only interested in the anatomy specific AE features for model regularisation.

The presented ACNN framework is not only limited to the medical image segmentation and SR tasks but can be extended to other image analysis tasks where prior knowledge can provide model guidance and robustness. In that regard, future research will focus on the application of ACNN to the problems such as human pose estimation, anatomical and facial landmark localisation on partially occluded image data.

V Acknowledgements

This study was supported by an EPSRC programme grant (EP/P001009/1) and by British Heart Foundation, UK, project grant (PG/12/27/29489). The research was also co-funded by the National Institute for Health Research (NIHR) Biomedical Research Centre based at Imperial College Healthcare NHS Trust. The views expressed are those of the author(s) and not necessarily those of the NHS, the EPSRC, the NIHR or the Department of Health.

VI Supplementary Material

VI-1 Implementation Details

In this section, we give the details about the meta-parameter choices and sampling strategies. In both the super-resolution (SR) and segmentation (Seg) models, the weight decay regularisation term is weighted by $\lambda_{2}=5\times 10^{-6}$, and gradient descent learning-rate is fixed to $lr=0.001$. The weight of global priors is chosen experimentally to be $\lambda_{1}=0.01$. Mini batch-size, which is the number of samples used for each back-propagated gradient update, is set to be $8$ samples. The models are trained with full images without a need for patch extraction since the cardiac 2D MR image stack size is relatively smaller compared to the available GPU memory (Nvidia GTX-1080).

In the SR problem, the through-plane upsampling factor is fixed to $K=5$ and the synthetic low-resolution training samples are generated simply by filtering high-resolution images with a Gaussian blurring kernel ($\sigma=4.0$ mm) along the through plane direction. The blurring operation is followed by a decimation operator along the same image dimension. In the segmentation problem, the Sorensen-Dice loss was tested in the experiments as an alternative to the cross-entropy loss, yet we observed a degraded performance since the latter is a smoother function.

VI-2 Network Structure

In this section, we give the network structures of the autoencoder (AE) and the predictor, which together build the T-L network. The details are provided in Table VI-2 and VI-2. As can be seen in the tables, residual connections are not used in our models ($10$-$18$ layers) since they do not provide significant accuracy gains for smaller networks as reported in [26]. Additionally, non-linear layers are not applied on the lower dimensional latent representations since that would further constrain the autoencoder. The number of the hidden units ($64$) is chosen experimentally, and optimal configurations can be explored to improve the reconstruction performance. The input layer of the AE model is extended to multi-label segmentation maps through one-hot image representation, where each label is converted into a separate channel at the input. Lastly, in both models (AE and PR) each convolution layer, except the last one, is followed by batch-normalisation for better convergence behaviour.