Semi-Supervised and Task-Driven Data Augmentation

TL;DR

This paper introduces a task-driven data augmentation method that explicitly models deformation and intensity variations using a generative network, improving segmentation performance especially with limited labeled data.

Contribution

The proposed method explicitly models deformation fields and intensity masks, leveraging unlabelled data, and optimizes the generative process for segmentation tasks.

Findings

Significantly outperforms conventional augmentation in small data scenarios

Effectively models shape and intensity variations

Utilizes unlabelled data to enhance augmentation quality

Abstract

Supervised deep learning methods for segmentation require large amounts of labelled training data, without which they are prone to overfitting, not generalizing well to unseen images. In practice, obtaining a large number of annotations from clinical experts is expensive and time-consuming. One way to address scarcity of annotated examples is data augmentation using random spatial and intensity transformations. Recently, it has been proposed to use generative models to synthesize realistic training examples, complementing the random augmentation. So far, these methods have yielded limited gains over the random augmentation. However, there is potential to improve the approach by (i) explicitly modeling deformation fields (non-affine spatial transformation) and intensity transformations and (ii) leveraging unlabelled data during the generative process. With this motivation, we propose a novel task-driven data augmentation method where to synthesize new training examples, a generative network explicitly models and applies deformation fields and additive intensity masks on existing labelled data, modeling shape and intensity variations, respectively. Crucially, the generative model is optimized to be conducive to the task, in this case segmentation, and constrained to match the distribution of images observed from labelled and unlabelled samples. Furthermore, explicit modeling of deformation fields allow synthesizing segmentation masks and images in exact correspondence by simply applying the generated transformation to an input image and the corresponding annotation. Our experiments on cardiac magnetic resonance images (MRI) showed that, for the task of segmentation in small training data scenarios, the proposed method substantially outperforms conventional augmentation techniques.

Tables

Table 1. Table 1: Average Dice score (DSC) results over 15 runs of 20 test subjects for the proposed method and relevant works. ∗ , † , ⋆ ∗ † ⋆ \ast,\dagger,\star denotes statistical significance over Aug A,RD , Aug A,RI abd Aug A,Mixup respectively. (Wilcoxon signed rank test with threshold p value of 0.05).

Method	Number of 3D training volumes used
	1			3
	RV	Myo	LV	RV	Myo	LV
Augnone	0.259	0.291	0.446	0.589	0.631	0.805
AugA	0.373	0.484	0.644	0.733	0.744	0.885
AugA,RD	0.397	0.503	0.663	0.756	0.763	0.897
AugA,GD(${\lambda }_{adv}=1,{\lambda }_{big}={10}^{-3}$)	${0.487}^{\ast }$	${0.560}^{\ast }$	${0.717}^{\ast }$	${0.782}^{\ast }$	${0.791}^{\ast }$	${0.908}^{\ast }$
AugA,GD(${\lambda }_{adv}=0,{\lambda }_{big}=0$)	0.394	${0.531}^{\ast }$	${0.694}^{\ast }$	0.756	${0.776}^{\ast }$	${0.908}^{\ast }$
AugA,RI	0.429	0.554	0.742	0.744	0.759	0.896
AugA,GI(${\lambda }_{adv}=1,{\lambda }_{big}={10}^{-3}$)	${0.517}^{†}$	${0.579}^{†}$	${0.773}^{†}$	${0.803}^{†}$	${0.791}^{†}$	0.912
AugA,GI(${\lambda }_{adv}=0,{\lambda }_{big}=0$)	${0.500}^{†}$	${0.583}^{†}$	${0.765}^{†}$	${0.766}^{†}$	${0.748}^{†}$	0.893
AugA,GD,GI(${\lambda }_{adv}=1,{\lambda }_{big}={10}^{-3}$)	${0.651}^{\star }$	${0.710}^{\star }$	${0.834}^{\star }$	${0.832}^{\star }$	${0.823}^{\star }$	${0.922}^{\star }$
AugA,Mixup [19]	0.581	0.599	0.774	0.818	0.791	0.915
AugA,GD,GI,Mixup	0.679	0.713	0.849	0.844	0.825	0.924
Adv Tr SL [12]	0.417	0.507	0.698	0.731	0.753	0.891
Adv Tr SSL [20]	0.409	0.506	0.692	0.692	0.719	0.874

Equations

$$\min_{w_{S}}\Big{(}L_{S}(\{X_{L},Y_{L}\})\Big{)}$$

$$\min_{w_{S}}\Big{(}L_{S}(\{X_{L},Y_{L}\}\cup\{X_{G},Y_{G}\})\Big{)}$$

$$\underset{w_G}{\min }\underset{w_D}{\max }{\mathbb{E}}_{x,y\sim p(x_L,y_L)}[\log D(x,y)]+{\mathbb{E}}_{z\sim p_z(z)}[\log (1-D(G(z))]$$

$$\min_{w_{G}}\Big{(}\min_{w_{S}}L_{S}(\{X_{L},Y_{L}\}\cup\{X_{G},Y_{G}\})+L_{reg,w_{G}}(\{X_{L}\}\cup\{X_{UL}\})\Big{)}$$

$$L_{reg,w_{G_C}}={\lambda }_{adv}{\mathbb{E}}_{z\sim p_z(z)}[\log (1-D_C(G_C(z,X_L)))]+{\lambda }_{big}{L}_{G_C,big}$$

$$L_{D_C}=-{\mathbb{E}}_{x\sim p(x)}[\log D_C(x)]-{\mathbb{E}}_{z\sim p_z(z)}[\log (1-D_C(G_C(z,X_L)))]$$

$$X_G=\lambda X_{Li}+(1-\lambda )X_{Lj},Y_G=\lambda Y_{Li}+(1-\lambda )Y_{Lj}$$

Full text

1 Introduction

††This article has been accepted at the 26th international conference on Information Processing in Medical Imaging (IPMI) 2019.

Precise segmentation of anatomical structures is crucial for several clinical applications. Recent advances in deep neural networks yielded automatic segmentation algorithms with unprecedented accuracy. However, such methods heavily rely on large annotated training datasets. In this work, we consider the problem of medical image segmentation in the setting of small training datasets.

Let us first consider the question: why is a large training dataset necessary for the success of deep learning methods? One hypothesis is that a large training dataset exposes a neural network to sufficient variations in factors, such as shape, intensity and texture, thereby allowing it to learn a robust image to segmentation mask mapping. In medical images, such variations may arise from subject specific shape differences in anatomy or lesions. Image intensity and contrast characteristics may differ substantially according to the image acquisition protocol or even between scanners for the same acquisition protocol. When the training dataset is small, deep learning methods are susceptible to faring poorly on unseen test images either due to not identifying such variations or because the test images appear to have been drawn from a distribution different to the training images.

We conjecture that one way to train a segmentation network on a small training dataset more robustly could be to incorporate into the training dataset, intensity and anatomical shape variations observed from a large pool of unlabelled images. Specifically, we propose to generate synthetic image-label pairs by learning generative models of deformation fields and intensity transformations that map the available labelled training images to the distribution of the entire pool of available images, including labelled as well as unlabelled. Additionally, we explicitly encourage the synthesized image-label pairs to be conducive to the task at hand. We carried out extensive evaluation of the proposed method, in which the method showed substantial improvements over existing data augmentation as well as semi-supervised learning techniques for segmentation of cardiac MRIs.

Related work: Due to the high cost of obtaining large amount of expert annotations, robust training of machine learning methods in the small training dataset setting has been widely studied in the literature. Focusing on the methods that are most relevant to the proposed method, we broadly classify the related works into two categories:

Data augmentation is a technique wherein the training dataset is enlarged with artificially synthesized image-label pairs. The main idea is to transform training images in such a way that the corresponding labels are either unchanged or get transformed in the same way. Some commonly used data augmentation methods are affine transformations [6] (such as translation, rotation, scaling, flipping, cropping, etc.) and random elastic deformations [17, 15]. Leveraging recent advances in generative image modelling [9], several works proposed to map randomly sampled vectors from a simple distribution to realistic image-label pairs as augmented data for medical image segmentation problems [4, 7, 16]. Such methods are typically trained on already labelled data, with the objective of interpolating within the training dataset. In an alternative direction, [19] proposed to synthesize data for augmentation by simply linearly interpolating the available images and the corresponding labels. Surprisingly, despite employing clearly unrealistic images, this method led to substantial improvements in medical image segmentation [8] when the available training dataset is very small. None of these data augmentation methods use unlabelled images that may be more readily available and all of them, except for those based on generative models, are hand-crafted rather than optimized based on data.

Semi-supervised learning (SSL) methods are another class of techniques that are suitable in the setting of learning with small labelled training datasets. The main idea of these methods is to regularize the learning process by employing unlabelled images. Approaches based on self-training [2] alternately train a network with labeled images, estimate labels for the unlabelled images using the network and update the network with both the available true image-label pairs and the estimated labels for the unlabelled images. [20] propose a SSL method based on adversarial learning, where the joint distribution of unlabelled image-estimated labels pairs is matched to that of the true labelled images-label pairs. Interestingly, [13] show that many SSL methods fail to provide substantial gains over the supervised baseline that is trained with data augmentation and regularization.

Weakly-supervised learning tackles the issue of expensive pixel-wise annotations by training on weaker labels, such as scribbles [5] and image-wide labels [1]. Finally, other regularization methods that do not necessarily leverage unlabelled images may also aid in preventing over-fitting to small training datasets.

2 Methods

In a supervised learning setup, an objective function $L_{S}(\{X_{L},Y_{L}\})$ that measures discrepancy between ground truth labels, $Y_{L}$, and predictions of a network $S$ on training images, $X_{L}$, is minimized with respect to a set of learnable parameters $w_{S}$ of the network, i.e.

[TABLE]

When data augmentation is employed in the supervised learning setup, Eq. 2 is minimized with respect to $w_{S}$.

[TABLE]

Here, $X_{G}$ and $Y_{G}$ refer to generated images and labels obtained by affine or elastic transformations of $X_{L}$, $Y_{L}$ or by using methods such as Mixup [19]. The set $\left\{\{X_{L},Y_{L}\}\cup\{X_{G},Y_{G}\}\right\}$ is referred to as the augmented training set. In augmentation methods based on generative models [4], the parameters of $S$ are still optimized according to Eq. 2, but the generative process for $X_{G}$ and $Y_{G}$ involves two other networks: a generator $G$ and a discriminator $D$. The corresponding parameters $w_{G}$ and $w_{D}$ are estimated according to the generative adversarial learning (GAN) [9] framework by optimizing:

[TABLE]

$G$ takes as input a vector $z$ sampled from a known distribution $p_{z}(z)$ and maps that to a {$X_{G}$, $Y_{G}$} pair. $D$ is optimized to distinguish between outputs of $G$ and real {$X_{L}$, $Y_{L}$} pairs, while $G$ is optimized to generate {$X_{G}$, $Y_{G}$} pairs such that $D$ responds to them similarly as to {$X_{L}$, $Y_{L}$}. Thus, $\{X_{G},Y_{G}\}$ are encouraged to be “realistic” in the sense that they cannot be distinguished by $D$.

2.1 Semi-Supervised and Task-Driven Data Augmentation

Instead of solving the optimization given in Eq. 3 for generating the augmentation image-label pairs {$X_{G}$, $Y_{G}$}, we propose solving Eq. 4:

[TABLE]

This incorporates two ideas. The first term dictates that {$X_{G}$, $Y_{G}$} be such that they are beneficial for minimizing the segmentation loss $L_{S}$. Secondly, note that $L_{reg,w_{G}}$ depends not only on the labelled images {$X_{L}$} (as in Eq. 3), but also on the unlabelled images {$X_{UL}$}. It is a regularization term based on an adversarial loss, which incorporates information about the image distribution that can be extracted from both {$X_{L}$} and {$X_{UL}$}. This is achieved by synthesizing $\{X_{G}\}$ as $G_{C}(\{X_{L}\})$, where $G_{C}$ denotes a conditional generative model. $G_{C}$ is modelled in two different ways: one for deformation field (non-affine spatial transformations) and one for intensity transformations. In both cases, the formulation is such that as a certain labelled image is mapped to an augmentation image, the mapping to obtain the corresponding augmentation label readily follows.

2.1.1 Deformation Field Generator

: The deformation field generator, $G_{C}$ = $G_{V}$, is trained to create samples from the distribution of deformation fields that can potentially map elements from $\{X_{L}\}$ to those in the combined set $\{X_{L}\}\cup\{X_{UL}\}$. $G_{V}$ takes as input an image from $\{X_{L}\}$ and a vector $z$, sampled from a unit Gaussian distribution, and outputs a dense per-pixel deformation field, v. The input image and its corresponding label (in 1-hot encoding) are warped using bilinear interpolation according to v to produce $X_{G,V}$ and $Y_{G,V}$ respectively.

2.1.2 Additive Intensity Field Generator

: The intensity field generator, $G_{C}$ = $G_{I}$, is trained to draw random samples from the distribution of additive intensity fields that can potentially map elements from $\{X_{L}\}$ to those in $\{X_{L}\}\cup\{X_{UL}\}$. $G_{I}$, takes as input an element of $\{X_{L}\}$ and a noise vector and outputs an intensity mask, $\Delta I$. $\Delta I$ is added to the input image to give the transformed image $X_{G,I}$, while its segmentation mask $Y_{G,I}$ remains the same as that of the input image.

2.1.3 Regularization Loss

: For both the conditional generators, the regularization term, $L_{reg,w_{G}}$, in Eq. 4 is formulated as in Eq. 5. The corresponding discriminator networks $D_{C}$ are trained to minimize the usual adversarial objective (Eq. 6).

[TABLE]

[TABLE]

The generated images are obtained as $G_{V}(z,X_{L})=\textbf{v}\circ X_{L}$ and $G_{I}(z,X_{L})=\Delta I+X_{L}$, where $\circ$ denotes a bilinear warping operation. In our experiments, we observe that with only the adversarial loss term in $L_{reg,w_{G_{C}}}$, the generators tend to create only the identity mapping. So, we introduce the $L_{G_{C},big}$ term to incentivize non-trivial transformations. We formulate $L_{G_{V},big}$ and $L_{G_{I},big}$ as $-||\textbf{v}||_{1}$ and $-||\Delta I||_{1}$ respectively.

2.1.4 Optimization Sequence

: The method starts by learning the optimal data augmentation for the segmentation task. Thus, all networks $S$, $G_{C}$ and $D_{C}$ are optimized according to Eq. 4. The generative models for the deformation fields and the intensity fields are trained separately. Once this is complete, both $G_{V}$ and $G_{I}$ are fixed and the parameters of $S$ are re-initialized. Now, $S$ is trained again according to Eq. 2, using the original labelled training data $\{X_{L},Y_{L}\}$ and augmentation data $\{X_{G},Y_{G}\}$ generated using the trained $G_{V}$ or $G_{I}$ or both.

3 Dataset and Network details

3.1 Dataset Details

We used a publicly available dataset hosted as part of MICCAI’17 ACDC challenge [3] 111https://www.creatis.insa-lyon.fr/Challenge/acdc. It comprises of short-axis cardiac cine-MRIs of 100 subjects from 5 groups - 20 normal controls and 20 each with 4 different cardiac abnormalities. The in-plane and through-plane resolutions of the images range from 0.70x0.70mm to 1.92x1.92mm and 5mm to 10mm respectively. Expert annotations are provided for left ventricle (LV), myocardiam (Myo) and right ventricle (RV) for both end-systole (ES) and end-diastole (ED) phases of each subject. For our experiments, we only used the ES images.

3.2 Pre-processing

We apply the following pre-processing steps to all images of the dataset: (i) bias correction using N4 [18] algorithm, (ii) normalization of each 3d image by linearly re-scaling the intensities as: $(x-x_{2})/(x_{98}-x_{2})$, where $x_{2}$ and $x_{98}$ are the $2^{nd}$ and $98^{th}$ percentile in the bias corrected 3d image, (iii) re-sample each slice of each 3d image and the corresponding labels to an in-plane resolution of 1.367x1.367mm using bi-linear and nearest neighbour interpolation respectively and crop or pad them to a fixed size of 224x224.

3.3 Network Architectures

There are three types of networks in the proposed method (see Fig. 1): a segmentation network $S$, a generator network $G$ and a discriminator network $D$. In this sub-section, we describe their architectures. Expect for the last layer of $G$, the same architecture is used for the $G_{V}$ and $G_{I}$ networks used for modelling both the deformation fields and the intensity transformations.

Generator: $G$ takes as input an image from $\{X_{L}\}$ and a noise vector $z$ of dimension 100, which are both first passed through separate sub-networks, $G_{subnet,X}$ and $G_{subnet,z}$. $G_{subnet,X}$, consists of 2 convolutional layers, while $G_{subnet,z}$, consists of a fully-connected layer, followed by reshaping of the output, followed by 5 convolutional layers, interleaved with bilinear upsampling layers. The outputs of the two sub-networks are of the same dimensions. They are concatenated and passed through a common sub-network, $G_{subnet,common}$, consisting of 4 convolutional layers, the last of which is different for $G_{V}$ and $G_{I}$. The final convolutional layer for $G_{V}$ outputs two feature maps corresponding to the 2-dimensional deformation field v, while that for $G_{I}$ outputs a single feature map corresponding to the intensity mask $\Delta I$. The final layer of $G_{I}$ employs the tanh activation to cap the range of the intensity mask. All other layers use the ReLU activation. All convolutional layers have 3x3 kernels except for the final ones in both $G_{V}$ and $G_{I}$ and are followed by batch-normalization layers before the activation.

Discriminator: $D$ consists of 5 convolutional layers with kernel size of 5x5 and stride 2. The convolutions are followed by batch normalization layers and leaky ReLU activations with the negative slope of the leak set to 0.2. After the convolutional layers, the output is reshaped and passed through 3 fully-connected layers, with the final layer having an output size of 2.

Segmentation Network: We use a U-net [15] like architecture for $S$. It has an encoding and a decoding path. In the encoder, there are 4 convolutional blocks, each consisting of 2 3x3 convolutions, followed by a max-pooling layer. The decoder consists of 4 convolutional blocks, each made of a concatenation with the corresponding features of the encoder, followed by 2 3x3 convolutions, followed by bi-linear upsampling with factor 2. Batch normalization and ReLU activation are employed in all layers, except the last one.

3.4 Training Details

Weighted cross-entropy is used as the segmentation loss, $L_{S}$. We empirically set the weights of the 4 output labels to 0.1 (background) and 0.3 (each of the 3 foreground labels). The background loss is considered while learning the augmentations, but not while learning the segmentation task alone. We empirically set $\lambda_{adv}$ and $\lambda_{big}$ to 1 and $10^{-3}$ respectively. The batch size is set to 20 and each training is run for 10000 iterations. The model parameters that provide the best dice score on the validation set are chosen for evaluation. Adam optimizer is used for all networks with an initial learning rate of $10^{-3}$, $\beta_{1}=0.9$ and $\beta_{2}=0.999$.

4 Experiments

We divide the dataset into test ($X_{ts}$), validation ($X_{vl}$), labelled training ($X_{L}$) and unlabelled training ($X_{UL}$) sets which consist of 20, 2, NL and 25 3d images respectively. As we are interested in the few labelled training images scenario, we run all our experiments in two settings: with NL set to 1 and 3. $X_{ts}$, $X_{vl}$ and $X_{UL}$ are selected randomly a-priori and fixed for all experiments. $X_{ts}$ and $X_{UL}$ are chosen such that they consist of equal number of images from each group (see Sec. 3.1) of the dataset. A separate set of 10 images (2 from each group), $X_{L,total}$, is selected randomly. Each experiment is run 5 times with $X_{L}$ as NL images randomly selected from $X_{L,total}$. When NL is 3, it is ensured that the images in $X_{L}$ come from different groups. Further, each of the 5 runs with different $X_{L}$ is run thrice in order to account for variations in convergence of the networks. Thus, overall, we have 15 runs for each experiment.

The following experiments were done thrice for each choice of $X_{tr,L}$:

• •

No data augmentation (Augnone): $S$ is trained without data augmentation.

• •

Affine data augmentation(AugA): $S$ is trained with data augmentation comprising of affine transformations. These consist of rotation (randomly chosen between -15deg and +15deg), scaling (with a factor randomly chosen uniformly between 0.9 and 1.1), another possible rotation that is multiple of 45deg (angle=45deg*N where N is randomly chosen between 0 to 8), and flipping along x-axis. For each slice in a batch, a random number between 0 and 5 is uniformly sampled and accordingly, either the slice is left as it is or is transformed by one of the 4 stated transformations.

All the following data augmentation methods, each training batch (batch size$=$bs) is first applied affine transformations as explained above. The batch used for training consists of half of these images along with bs/2 augmentation images obtained according to the particular augmentation method.

• •

Random elastic deformations (AugA,RD): Elastic augmentations are modelled as in [15], where a deformation field is created by sampling each element of a 3x3x2 matrix from a Gaussian distribution with mean 0 and standard deviation 10 and upscaling it to the image dimensions using bi-cubic interpolation.

• •

Random contrast and brightness fluctuations [11, 14] (AugA,RI): This comprises of an image contrast adjustment step: $x=(x-\bar{x})*c+\bar{x}$, followed by a brightness adjustment step: $x=x+b$. We sample c and b uniformly in [0.8,1.2] and [-0.1,0.1] respectively.

• •

Deformation field transformations (AugA,GD): Augmentation data is generated from the trained deformation field generator $G_{V}$.

• •

Intensity field transformations (AugA,GI): Augmentation data is generated from the trained intensity field generator $G_{I}$.

• •

Both deformation and intensity field transformations (AugA,GD,GI): In this experiment, we sample data from $G_{V}$ and $G_{I}$ to obtain transformed images $X_{V}$ and $X_{I}$ respectively. We also get an additional set of images which contain both deformation and intensity transformations $X_{VI}$. These are obtained by conditioning $G_{I}$ on spatially transformed images $X_{V}$. The augmentation data comprises of all these images $\{X_{V},X_{I},X_{VI}\}$.

• •

MixUp [19] (AugA,Mixup): Augmentation data ($\{X_{G},Y_{G}\}$) is generated using the original annotated images $X_{L}$ and their linear combinations using the Mixup formulation as stated in Eq. 7.

[TABLE]

where $\lambda$ is sampled from beta distribution Beta$(\alpha,\alpha)$ with $\alpha\in(0,\infty)$ and $\lambda\in[0,1)$ which controls the ratio to mix the image-label pairs $(X_{Li},Y_{Li})$, $(X_{Lj},Y_{Lj})$ selected randomly from the set of labelled training images.

• •

**Mixup over deformation and intensity field transformations

(AugA,GD,GI,Mixup)**: Mixup is applied over different pairs of available images: original data ($X_{L}$), their affine transformations and the images generated using deformation and intensity field generators $\{X_{V},X_{I},X_{VI}\}$.

• •

Adversarial Training (Adv Tr): Here, we investigate the benefit of the method proposed in [20] on our dataset (explained in Sec. 1), in both supervised (SL) [12] and semi-supervised (SSL) [20] settings.

Evaluation : The segmentation performance of each method is evaluated using the Dice similarity coefficient (DSC) over 20 test subjects for three foreground structures: left ventricle (LV), myocardiam (Myo) and right ventricle (RV).

5 Results and Discussion

Table 1 presents quantitative results of our experiments. The reported numbers are the mean dice scores over the 15 runs for each experiments as described in Sec. 4. It can be observed that the proposed method provides substantial improvements over other data augmentation methods as well as the semi-supervised adversarial learning method, especially in the case where only 1 3D volume is used for training. The improvements can also be visually observed in Fig. 2. In the rest of this section, we discuss the results of specific experiments.

Perhaps unsurprisingly, the lowest performance occurs when neither data augmentation nor semi-supervised training is used. Data augmentation with affine transformations already provides remarkable gains in performance. Both random elastic deformations and random intensity fluctuations further improve accuracy.

The proposed augmentations based on learned deformation fields improve performance as compared to random elastic augmentations. These results show the benefit of encouraging the deformations to span the geometric variations present in entire population (labelled as well as unlabelled images), while still generating images that are conducive to the training of the segmentation network. Some examples of the generated deformed images are shown in Fig. 3. Interestingly, the anatomical shapes in these images are not always realistic. While this may appear to be counter-intuitive, perhaps preserving realistic shapes of anatomical structures in not essential to obtaining the best segmentation neural network.

Similar observations can be made about the proposed augmentations based on learned additive intensity masks as compared to random intensity fluctuations. Again, the improvements may be attributed to encouraging the intensity transformations to span the intensity statistics present in the population, while being beneficial for the segmentation task. Qualitatively, also as before, the generated intensity masks (Fig. 3) do not necessarily lead to realistic images.

As both $G_{V}$ and $G_{I}$ are designed to capture different characteristics of the entire dataset, using both the augmentations together may be expected to provide a higher benefit than employing either one in isolation. Indeed, we observe a substantial improvement in dice scores with our experiments.

As an additional experiment, we investigated the effect of excluding the regularization term from the training of the generators, $G_{V}$ and $G_{I}$ ($\lambda_{adv}=\lambda_{big}=0$). While the resulting augmentations still resulted in better performance than random deformations or intensity fluctuations, their benefits were lesser than that from the ones that were trained with the regularization. This shows that although the adversarial loss does not ensure the generation of realistic images, it is still advantageous to include unlabelled images in the learning of the augmentations.

Augmentations obtained from the Mixup [19] method also lead to a substantial improvement in performance as compared to using affine transformations, random elastic transformations or random intensity fluctuations. Interestingly, this benefit also occurs despite the augmented images being not realistic looking at all. One reason for this behaviour might be that the Mixup augmentation method provides soft probability labels for the augmented images - such soft targets have been hypothesized to aid optimization by providing more task information per training sample [10]. Even so, Mixup can only generate augmented images that are linear combinations of the available labelled images and it is not immediately clear how to extend this method to use unlabelled images. Finally, we see that Mixup provides a marginal improvement when applied over the original images together with the augmentations obtained from the trained generators $G_{V}$ and $G_{I}$. This demonstrates the complementary benefits of the two approaches.

While semi-supervised adversarial learning provides improvement in performance as compared to training with no data augmentation, these benefits are only as much as those obtained with simple affine augmentation. This observation seems to be in line with works such as [13].

6 Conclusion

One of the challenging requirements for the success of deep learning methods in medical image analysis problems is that of assembling large-scale annotated datasets. In this work, we propose a semi-supervised and task-driven data augmentation approach to tackle the problem of robust image segmentation in the setting of training datasets consisting of as few as 1 to 3 labelled 3d images. This is achieved via two novel contributions: (i) learning conditional generative models of mappings between labelled and unlabelled images, in a formulation that also readily enables the corresponding segmentation annotations to be appropriately mapped and (ii) guiding these generative models with task-dependent losses. In the small labelled data setting, for the task of segmenting cardiac MRIs, we show that the proposed augmentation method substantially outperforms the conventional data augmentation techniques. Interestingly, we observe that in order to obtain improved segmentation performance, the generated augmentation images do not necessarily need to be visually hyper-realistic.