Patch-based Output Space Adversarial Learning for Joint Optic Disc and Cup Segmentation

TL;DR

This paper introduces pOSAL, a patch-based adversarial learning framework that improves joint optic disc and cup segmentation across different datasets by addressing domain shift and capturing local segmentation details.

Contribution

The paper proposes a novel patch-based adversarial learning framework with a morphology-aware loss for robust, cross-dataset optic disc and cup segmentation.

Findings

Improved segmentation performance on three public datasets.

Achieved first place in MICCAI 2018 Retinal Fundus Glaucoma Challenge.

Effective domain adaptation for retinal image segmentation.

Abstract

Glaucoma is a leading cause of irreversible blindness. Accurate segmentation of the optic disc (OD) and cup (OC) from fundus images is beneficial to glaucoma screening and diagnosis. Recently, convolutional neural networks demonstrate promising progress in joint OD and OC segmentation. However, affected by the domain shift among different datasets, deep networks are severely hindered in generalizing across different scanners and institutions. In this paper, we present a novel patchbased Output Space Adversarial Learning framework (pOSAL) to jointly and robustly segment the OD and OC from different fundus image datasets. We first devise a lightweight and efficient segmentation network as a backbone. Considering the specific morphology of OD and OC, a novel morphology-aware segmentation loss is proposed to guide the network to generate accurate and smooth segmentation. Our pOSAL framework then exploits unsupervised domain adaptation to address the domain shift challenge by encouraging the segmentation in the target domain to be similar to the source ones. Since the whole-segmentationbased adversarial loss is insufficient to drive the network to capture segmentation details, we further design the pOSAL in a patch-based fashion to enable fine-grained discrimination on local segmentation details. We extensively evaluate our pOSAL framework and demonstrate its effectiveness in improving the segmentation performance on three public retinal fundus image datasets, i.e., Drishti-GS, RIM-ONE-r3, and REFUGE. Furthermore, our pOSAL framework achieved the first place in the OD and OC segmentation tasks in MICCAI 2018 Retinal Fundus Glaucoma Challenge.

Tables

Table 1. TABLE I: Statistics of the datasets used in evaluating the proposed method.

Domain	Dataset	Number of samples	Image size	Cameras	Release year
Source	REFUGE Train	400	$2124\times 2056$	Zeiss Visucam 500	2018
Target	Drishti-GS Train/Test	50 + 51	$2047\times 1759$	unknown	$2014$
Target	RIM-ONE-r3 Train/Test	99 + 60	$2144\times 1424$	unknown	$2015$
Target	REFUGE Validation/Test	400 + 400	$1634\times 1634$	Canon CR-2	$2018$

Table 2. TABLE II: Results of joint OD and OC segmentation on the Drishti-GS and RIM-ONE-r3 testing datasets.

Drishti-GS				RIM-ONE-r3
Method	$D{I}_{cup}$	$D{I}_{disc}$	$\delta$	Method	$D{I}_{cup}$	$D{I}_{disc}$	$\delta$
pOSAL	0.858	0.965	0.082	pOSAL	0.787	0.865	0.081
pOSALseg-S	0.836	0.944	0.118	pOSALseg-S	0.744	0.779	0.103
Edupuganti et al.[29]	0.897	0.967	-	DRIU [4]	-	0.955	-
Sevastopolsky [27]	0.850	-	-	Sevastopolsky [27]	-	0.950	-
Son et al.[43]	-	0.967	-	Son et al.[43]	-	0.955	-
Zilly et al.[26]	0.871	0.973	-	Zilly et al.[26]	0.824	0.942	-
pOSALseg-T	0.901	0.974	0.048	pOSALseg-T	0.856	0.968	0.049

Table 3. TABLE III: Comparison with different domain adaptation methods on the Drishti-GS and RIM-ONE-r3 datasets.

Method	Drishti-GS			RIM-ONE-r3
	$D{I}_{cup}$	$D{I}_{disc}$	$\delta$	$D{I}_{cup}$	$D{I}_{disc}$	$\delta$
TD-GAN[32]	0.747	0.924	0.117	0.728	0.853	0.118
Hoffman et al. [48]	0.851	0.959	0.093	0.755	0.852	0.082
Javanmardi et al. [31]	0.849	0.961	0.091	0.779	0.853	0.085
OSAL-pixel	0.851	0.962	0.089	0.778	0.854	0.084
pOSAL (ours)	0.858	0.965	0.082	0.787	0.865	0.081

Table 4. TABLE IV: Results of OD and OC segmentation on the REFUGE testing dataset. Top three items are bold for each metric.

Team	$D{I}_{cup}$	$R_{D{I}_{cup}}$	$D{I}_{disc}$	$R_{D{I}_{disc}}$	$\delta$	$R_{\delta }$	$S_f$	Overall Rank
CUHKMED (ours)	0.8826	2	0.9602	1	0.0450	2	1.75	1
Masker	0.8837	1	0.9464	7	0.0414	1	2.50	2
BUCT	0.8728	3	0.9525	3	0.0456	3	3.00	3
NKSG	0.8643	5	0.9488	5	0.0465	4	4.60	4
VRT	0.8600	6	0.9532	2	0.0525	7	5.40	5
AIML	0.8519	7	0.9505	4	0.0469	5	5.45	6
Mammoth	0.8667	4	0.9361	10	0.0526	8	7.10	7
SMILEDeepDR	0.8367	8	0.9386	9	0.0488	6	7.45	8
NIGHTOwl	0.8257	10	0.9487	6	0.0563	9	8.60	9
SDSAIRC	0.8315	9	0.9436	8	0.0674	10	9.15	10
Cvblab	0.7728	11	0.9077	11	0.0798	11	11.00	11
Winter_Fell	0.6861	12	0.8772	12	0.1536	12	12.00	12

Table 5. TABLE V: Results of OD and OC segmentation on the REFUGE validation dataset.

Method	$D{I}_{cup}$	$D{I}_{disc}$	$\delta$
pOSALseg-S	0.869	0.932	0.059
pOSAL	0.875	0.946	0.051

Table 6. TABLE VI: Comparison of different loss functions.

Loss function	$D{I}_{cup}$	$D{I}_{disc}$
Cross Entropy Loss	0.860	0.953
Dice Loss	0.878	0.950
Morphology-aware Loss	0.885	0.956

Table 7. TABLE VII: Comparison of different network backbones.

Block type	Params	$D{I}_{cup}$	$D{I}_{disc}$	Time
Xception	41.3M	0.885	0.953	0.124s
MobileNetV2	5.8M	0.885	0.956	0.056s

Table 8. TABLE VIII: Performance of the Extraction Network E .

Method	Drishti-GS			RIM-ONE-r3
	$D{I}_{cup}$	$D{I}_{disc}$	$\delta$	$D{I}_{cup}$	$D{I}_{disc}$	$\delta$
E	0.798	0.930	0.098	0.638	0.709	0.150
pOSAL	0.858	0.965	0.082	0.787	0.865	0.081

Equations

$$L_{DL}(p,y)=1-\frac{2{\sum }_{i\in \Omega }{p}_i\cdot y_i}{{\sum }_{i\in \Omega }{p}_i^2+{\sum }_{i\in \Omega }{y}_i^2},$$

$$L_{SL}(p,y)$$

$$B_{i,j}$$

$$L_{seg}$$

$$+{\lambda }_3[L_{SL}(p^d,y^d)+L_{SL}(p^c,y^c)],$$

$$L_D=-\sum _{m,n}z\log (D(S(I_{cS})))+(1-z)\log (1-D(S(I_{cT}))),$$

$$L_S=L_{seg}(I_{cS})+L_{adv}(I_{cT}),$$

$$L_{adv}(I_{cT})=-\sum _{m,n}\log (D(S(I_{cT}))).$$

$$DI=\frac{2\times N_{TP}}{2\times N_{TP}+N_{FP}+N_{FN}},$$

$$\delta =|CD{R}_p-CD{R}_g|,CDR=\frac{V{D}_{cup}}{V{D}_{disc}},$$

$${\hat{p}}_i=\frac{p_i-p_{min}}{p_{max}-p_{min}},$$

$$S_f=0.35\times R_{D{I}_{cup}}+0.25\times R_{D{I}_{disc}}+0.4\times R_{\delta },$$

Full text

I Introduction

Glaucoma is a chronic disease that damages the optic nerves and leads to irreversible vision loss [2]. Screening and detecting glaucoma in its early stage are beneficial to preserve the vision of patients. Currently, analyzing the optic nerve head and retinal nerve fiber layer is a practical method for glaucoma detection. However, such analysis is predominantly subjective and often suffers from the high intra- and inter-observer variations [3]. With the recent advancement in optical fundus imaging, objective and quantitative glaucoma assessments based on the morphology of optic disc (OD) and optic cup (OC), and the cup-to-disc ratio (CDR) become available [2]. CDR is the ratio of vertical cup diameter to vertical disc diameter. A large CDR value often indicates a high risk of glaucoma. Manually acquiring those measurements is time-consuming. Accurately segmenting OD and OC from the fundus image via automatic solutions would prompt the large-scale glaucoma screening [1].

Remarkable performance on OD and OC segmentation are recently reported with the development of deep learning [4, 5, 1]. Assuming the training and testing samples have the same appearance distribution, training dataset consisting of a large amount of pixel-level annotations helps the deep networks learn the segmentation on the testing dataset. However, it is difficult for the network to obtain good segmentation performance on new datasets. For example, the state-of-the-art network, like M-Net [1] performs well on its specific testing dataset, i.e., ORIGA [6], but generalizes poorly on some other datasets; see Fig. 1. Domain shift, which refers to the difference in appearance distribution between the different datasets, is the main cause for the poor generalization ability of deep networks [7, 8, 9]. Indeed, domain shift among various retinal fundus image datasets is very common. Many public retinal image datasets, e.g., Drishti-GS [10], RIM-ONE-r3 [11], and REFUGE, are acquired with obvious appearance discrepancy resulting from different scanners, image resolution ratios, light source intensities, and parameter settings (Fig. 1). Overcoming the domain shift is highly desired to enhance the robustness of deep networks.

To reduce the performance degradation caused by domain shift, domain adaptation methods [8, 12] are developed to generalize the deep networks trained in a source domain to work more effectively in some other target domains with varying appearance. A vanilla solution is to fine-tune the segmentation network with a full-supervision provided by a large quantity of annotated samples from the target domain. However, preparing for the extra annotations in the target domain is highly time-consuming and expensive, and often suffers from inter-observer variations; moreover, such a solution is impractical for large-scale glaucoma screening. Therefore, an unsupervised domain adaptation approach without requiring extra annotations is highly desirable in real clinical scenarios. Furthermore, leveraging the knowledge shared across different domains can help the deep networks maintain their performance under various imaging conditions. For this joint OD and OC segmentation task, spatial and morphological structures in the output space (i.e., segmentation mask) are shared by different datasets and thus are beneficial to the mask prediction. For example, the OC is always contained inside the OD region, while both the OC and OD have ellipse-like shapes. Such spatial correlation information is crucial for domain adaptation but is typically ignored by existing deep-network-based segmentation methods.

In this work, we aim at jointly segmenting the OD and OC in retinal fundus images from different domains by introducing a novel patch-based Output Space Adversarial Learning framework (pOSAL). As the core workhorses in the framework, the lightweight network architecture for efficacy and the unsupervised domain adaptation for domain-invariance contribute to our promising performance. Our framework explores the annotated source domain images and unannotated target domain images to reduce the performance degradation on target domain. We first develop a representative segmentation network equipped with a morphology-aware segmentation loss to produce compelling segmentations. Effectively combining the designs of DeepLabv3+ [13] and depth-wise separable convolutional network MobileNetV2 [14], our segmentation network achieves a good balance between extracting multi-scale discriminative context features and computational burden. The proposed morphology-aware segmentation loss further guides the network to capture mask smoothness priors and therefore improves the segmentation. To overcome the domain shift challenge, inspired by [15], we adopt the output space adversarial learning via utilizing the spatial and morphological structures of the segmentation mask. Specifically, we attach a discriminator network to learn the abstract spatial and shape information from the label distributions of the source domain, and then employ the adversarial learning procedure to encourage the segmentation network to generate consistent predictions in a shared output space (e.g., the similar spatial layout and structure context) for the images in both source and target domains. Since the whole-segmentation-based adversarial scheme is weak in capturing segmentation details, we devise a patch-wise discriminator to capture the local statistics of the output space and guide the segmentation network to focus on the local structure similarity in the image patches. We extensively evaluate our pOSAL framework for the joint OD and OC segmentation on three public fundus image datasets (Drishti-GS, RIM-ONE-r3, and REFUGE). The pOSAL framework achieves state-of-the-art results, bringing significant improvements with the proposed patch-based output space adversarial learning.

Our main contributions are summarized as follows:

We exploit unsupervised domain adaptation for joint OD and OC segmentation over different retinal fundus image datasets. The presented novel pOSAL framework enables patch-based output space domain adaptation to reduce the segmentation performance degradation on target datasets with domain shift. 2. 2.

We design an efficient segmentation network equipped with a new morphology-aware segmentation loss to produce plausible OD and OC segmentation results. The morphological segmentation loss is able to guide the network to capture the mask smoothness priors for accurate segmentation. 3. 3.

We conduct extensive experiments on three public retinal fundus image datasets to demonstrate the effectiveness of the pOSAL framework. Furthermore, we achieved the first place in the OD and OC segmentation task of the MICCAI 2018 Retinal Fundus Glaucoma Challenge.

The remainders of this paper are organized as follows. We review the related techniques in Section II and elaborate the pOSAL framework in Section III. The experiments and results are presented in Section IV. We further discuss our method in Section V and draw the conclusions in Section VI.

II Related Works

The OD and OC segmentation from retinal fundus images are non-trivial and have been independently studied for years. For the OD segmentation, early works employed the hand-crafted visual features, including the image gradient information [16], features from stereo image pairs [17], local texture features [18] and superpixel-based classifier [19]. The OC segmentation is more challenging than OD considering the lower-contrast boundary [1]. Hand-crafted features are also investigated for this task [18, 19, 20, 21, 22, 23]. Recently, some works were developed for joint OD and OC segmentation. Zheng et al. [24] designed a graph-cut framework. In [25], structure constraints were utilized for joint OD and OC segmentation.

Convolutional neural networks (CNNs) have shown remarkable performance on retinal fundus image segmentation [1, 4, 5, 26, 27, 28, 29], and outperformed traditional hand-crafted features based methods [30]. Effective network architecture design is the focus of these deep learning based methods. For example, Maninis et al. [4] presented the DRIU network combining multi-level features to segment vessels and optic disc. A disc-aware network [28] was designed for glaucoma screening by an ensemble of different feature streams in the network. ResU-net was presented in [5] with an adversarial module between the ground truth and segmentation mask to improve the final segmentation. Based on the U-net, Fu et al. [1] developed the M-Net for joint OD and OC segmentation. Although promising, CNN-based methods are often degraded when the training and testing datasets are from different domains. Our output space adversarial learning framework helps address this domain shift challenge and enhance the segmentation performance on different testing domains.

Very recently, domain adaptation techniques were explored in the field of medical image analysis [8, 9, 31, 32, 33]. Previous methods [8, 9] performed the latent feature alignment to explore a shared feature space on the source and target domain through the adversarial learning. Another cut-in point for domain adaptation is to transfer the images from the target domain to the source domain, and then to apply the trained network to the transferred images [32, 34, 35]. Among these methods, Cycle-GAN [36] is a popular technique to transfer images over different domains. The key characteristic of these approaches is to generate style-realistic images in another domain without using paired data. Extra constraints are needed to guide this unsupervised style transfer process. For example, Zhang et al. [34] employed two segmentation networks stacked behind the cycle-GAN to act as an extra supervision on the generators to enhance the shape-consistency. In [35], a semantic-aware adversarial learning was introduced to prevent the semantic distortion during the image transformation. In [32], a task-driven generative adversarial network was developed to enforce the segmentation consistency. However, these methods ignore the property that for segmentation tasks, the label space (output space) of different domains are usually highly correlated in terms of the spatial structures and geometry. Therefore, instead of exploring a shared feature space or transferring the input images, we use a patch-based output space adversarial learning to conduct the domain adaptation for joint OD and OC segmentation.

III Methodology

Fig. 2 overviews the pOSAL framework for joint OD and OC segmentation from retinal fundus images; our framework has three modules: an ROI extraction network $E$, a segmentation network $S$, and a patch-level discriminator $D$. Due to the small area ratio of OD over the whole image, the ROI regions, $I_{cS}$ and $I_{cT}$, are firstly extracted from the source domain images $I_{S}$ and target domain images $I_{T}$, respectively (Section III-A). Then, the cropped source and target images $I_{cS}$ and $I_{cT}$ are fed into the segmentation network $S$ to produce the OD and OC predictions (Section III-B). A patch-level discriminator $D$ is utilized to encourage the segmentation network to produce similar outputs for the source domain images $I_{cS}$ and target domain images $I_{cT}$ (Section III-C). The whole framework is finally optimized by adversarial learning.

III-A ROI Extraction

To perform accurate segmentation, we first locate the position of OD and then crop the disc region from the original image for further segmentation. To achieve that, we build an extraction network $E$ to segment the OD and crop the ROI image according to the segmentation result. The extraction network is configured to segment the optic disc to provide a rough guidance. Although only trained with the source domain images and labels, as our experiments will demonstrate, the trained extraction network generalizes well on the target domain images due to the strong and visible structure characteristics of the optic disc in both source and target domain images. Therefore, the disc regions of both domain images can be obtained by the same extraction network. Specifically, our extraction network follows a U-Net [37] architecture and is trained with resized source images ($640\times 640$) and corresponding OD labels. The trained U-net can be used for coarse OD prediction in both domains. We then map the predicted OD mask back to the original image and crop a sub-image with the size of $512\times 512$ based on the center of the predicted OD mask. The extraction network $E$ has $19$ convolutional layers, and the last one is a $1\times 1$ convolutional layer with one output feature channel for the OD segmentation. We use the $Sigmoid$ activation function to generate the probability map of OD.

III-B Segmentation Network with Morphology-aware Loss

We conduct the OD and OC segmentation based on the above-cropped ROI images. To better capture the geometric structure of the output space, we customize a network with a novel morphology-aware segmentation loss for high-quality segmentation of the OD and OC.

III-B1 Segmentation Network Architecture

our segmentation network follows the spirit of DeepLabv3+ architecture [13]. To further reduce the number of parameters and the computation cost, we replace the backbone network Xception [13] with the lightweight and handy MobileNetV2 [14] as shown in Fig. 3. The first initial convolutional layer and the following seven inverted residual blocks of the MobileNetV2 are utilized to extract features. We keep the stride of the first convolutional layer and the following three blocks as the initial setting and set the stride as one in the remaining blocks. The total downsampling rate of the network is eight. The ASPP component [13] with different dilation rates is utilized to generate multi-scale features. The feature maps are then concatenated and followed by a $1\times 1$ convolutional layer. To integrate the semantic clues from different levels, we upsample the above-combined feature and concatenate it with the low-level feature for fine-grained semantic segmentation as the DeepLabv3+ does. Finally, we use another $1\times 1$ convolutional layer with two output channels followed by the $Sigmoid$ activation function to generate the probability maps of OD ($p^{d}$) and OC ($p^{c}$) simultaneously, according to the multi-label setting in [1]. The input size of the designed segmentation network is $512\times 512\times 3$, so that it can take the whole cropped images as input.

III-B2 Morphology-aware Segmentation Loss

to improve the segmentation, we develop a novel morphology-aware segmentation loss to guide the network to segment and capture the smoothness priors of the OD and OC. This joint morphological loss includes a dice coefficient loss $L_{DL}$ and a smoothness loss $L_{SL}$.

The dice coefficient loss [38] measures the overlap between the prediction and ground truth, and is written as

[TABLE]

where $\Omega$ are the total pixels in the image; $p$ and $y$ are the predicted probability map and binary ground truth mask, respectively.

The smoothness loss encourages the network to produce homogeneous predictions within neighbor regions. It is calculated by a binary pairwise label interaction:

[TABLE]

where $\mathcal{N}^{i}$ is the four-connected neighbors of pixel $i$; $p$ and $y$ denote the prediction and ground truth, respectively. The smoothness loss encourages the neighboring pixels $j$ of central pixel $i$ to have similar predicted probabilities when their ground truth belong to the same class ($B_{i,j}=1$). Smoothness loss is applied to the OD and OC probability maps, respectively.

The joint morphology-aware segmentation loss is defined as

[TABLE]

where $p^{d}$, $p^{c}$, $y^{d}$, $y^{c}$ are the predicted probability map and binary ground truth mask of OD and OC, respectively; $\lambda_{1},\lambda_{2},\lambda_{3}$ are the weights empirically set as 0.4, 0.6, and 1.0, respectively. Observing that it is more difficult to segment OC than OD due to the unclear boundaries of OC, we thus empirically set a slight larger value for $\lambda_{2}$ than $\lambda_{1}$.

III-C Patch-based Output Space Adversarial Learning

Different from high-level feature-based image classification, the feature for segmentation needs to encode both the low-level descriptors and high-level abstracts, such as appearance, shape, context and object semantic information. However, domain adaption based on feature space may not be the best choice for our segmentation task due to the complexity in handling the high-dimensional features [15]. Although the image appearance shifts across domains, the segmentation of source and target domain images have similar geometry structures in the output space (i.e., segmentation mask). Therefore, bridging two domains by forcing them to share the same distribution in the output space becomes an effective way for domain adaptation. In this work, we propose to perform domain adaptation for segmentation task through the output space adversarial learning. Specifically, the segmentation masks of target domain images should be similar to the ones of source domain. To achieve this, we attach a patch-level discriminator $D$ after the outputs of the segmentation network $S$, and then employ the adversarial learning technique to train the whole framework. In this adversarial setting, the segmentation network $S$ aims to fool the discriminator $D$ by generating a similar output space distribution either for source and target domain, while the discriminator aims to identify the segmentation from target domain as outliers. The geometry structure constraints on the segmentation masks are guaranteed through this adversarial process.

III-C1 Patch Discriminator

we employ a patch discriminator (PatchGAN) [39, 40] to conduct the adversarial learning. PatchGAN tries to classify whether each $m\times n$ overlapped patch from the predicted mask is in line with the distribution of that from the source predictions. Compared with the image-level (ImageGAN) or pixel-level (PixelGAN) adversarial learning, PatchGAN has the ability to capture the local statistics [41] of the output space and guides the segmentation network to focus on the local structure similarity in the image patches.

We realize the patch-based discriminator through a fully convolutional network, as shown in Fig. 4. The network contains five convolutional layers with a kernel size of $4\times 4$ and a stride of $2\times 2$. The channel number of the five convolutional layers are $64,128,256,512,1$, respectively. The activation function following each convolutional layer is LeakyReLU with an alpha value of $0.2$, except for the last one using the $Sigmoid$ function. The output size ($m\times n$) of the patch-based discriminator is $16\times 16$, in which one pixel corresponds to a patch of size $94\times 94$ in the input probability maps. Each patch is classified into real (1) or fake (0) through the discriminator. We employ this adversarial learning strategy to force each generated patch in the prediction of target domain to be similar to the patch of source domain.

III-C2 Objective Function

with the adversarial learning, we model the optimization as a two-player min-max game to alternately update weights in the segmentation network $S$ and the discriminator $D$.

The discriminator evaluates whether the input is from the source domain prediction. We formulate the training objective for the discriminator as

[TABLE]

where $z=1$ if the patch prediction is from the source domain, and $z=0$ if the patch prediction is from the target domain.

As for the segmentation network, the objective function consists of the proposed morphology-aware segmentation loss for the source domain images and the adversarial loss for the target domain images. In general, the training objective of segmentation network is

[TABLE]

Since we have the annotations for the images from source domain, we can use the joint morphology-aware segmentation loss $L_{seg}$ to optimize the segmentation network. The adversarial loss $L_{adv}$ is designed for the images in target domain $I_{cT}$ without any annotations. The segmentation network is responsible for ‘fooling’ the discriminator $D$ to classify the prediction of target domain images as the source prediction.

III-C3 Training Strategy

we optimize the segmentation network and the discriminator following the standard approach from [42]. In each training iteration, we feed the images from source domain $I_{cS}$ and target domain $I_{cT}$ to the network alternatively. Then we optimize the whole framework by minimizing the proposed objective functions $L_{S}$ and $L_{D}$. We repeat the above procedure for each training iteration.

IV Experiments

IV-A Dataset

We conducted experiments on three public OD and OC segmentation datasets, Drishti-GS dataset [10], RIM-ONE-r3 dataset [11] and the REFUGE challenge dataset111https://refuge.grand-challenge.org/Home/. The statistics of these three datasets are listed in Table I. We refer the train part of the REFUGE dataset as the source domain, the Drishti-GS dataset, RIM-ONE-r3 dataset and the validation/test parts of the REFUGE dataset as the target domain. The source and target domain images are captured by different cameras so that the color and texture of the images are different, as shown in Fig. 5. We first extensively evaluated and analyzed our pOSAL framework on the Drishti-GS and RIM-ONE-r3 datasets, and then compared with other state-of-the-art segmentation methods on the REFUGE test dataset.

IV-B Implementation Details

The framework was implemented in Python based on Keras [44] with the Tensorflow backend. We first trained the segmentation network with source domain images and annotations and then utilized the adversarial learning to train the whole pOSAL framework in an end-to-end manner. When training the segmentation network, we used the Adam [45] optimizer and initialized the backbone network weights by the MobileNetV2 [14] weights trained on the ImageNet dataset. We set the initial learning rate as $1e-3$ and divided it by $0.2$ every 100 epochs. We totally trained 200 epochs with a mini-batch size of 16 on a server with four Nvidia Titan Xp GPUs. Data augmentation was adopted to expand the training dataset by random scale, rotation, flip, elastic transformation, contrast adjustment, adding noise and random erasing [46]. When end-to-end training the whole pOSAL framework, we fed source and target images to the network alternatively. The optimization method of segmentation network was the same as the above, while the discriminator $D$ was optimized with the stochastic gradient descent (SGD) algorithm. The initial learning rate of the segmentation network and discriminator were $2.5e-5$ and $1e-5$, respectively, and decreased using the polynomial decay with a power of 0.9 as mentioned in [47] in a total of 100 epochs. We conducted the morphological operation, i.e., filling the hole, to post-process the predicted mask. The implementation and segmentation results are available in https://emmaw8.github.io/pOSAL.

IV-C Evaluation Metrics

We adopt the REFUGE challenge evaluation metrics, dice coefficients ($DI$) and the vertical cup to disc ratio ($CDR$), to evaluate the segmentation performance of the presented method. The criteria are defined as

[TABLE]

where $N_{TP}$ , $N_{FP}$, and $N_{FN}$ represent the number of true positive, false positive, and false negative pixels, respectively. $CDR_{p}$ and $CDR_{g}$ denote the cup to disc ratio value for the prediction and ground truth, while $VD_{cup}$ and $VD_{disc}$ are the vertical diameters for OC and OD, respectively. The dice coefficient is a standard evaluation metric for segmentation tasks, while the CDR value is one of the critical indicators for glaucoma screening in the clinical convention. We use absolute error $\delta$ to evaluate the difference between the CDR value of prediction $CDR_{p}$ and that of the ground truth $CDR_{g}$, while the lower $\delta$ value represents the better prediction result.

IV-D Experiments on Drishti-GS and RIM-ONE-r3 Datasets

Under the domain adaptation setting, we need to utilize the unlabeled target domain images to train the whole framework. For a fair comparison, the unlabeled target domain images used in the training phase were different from the target domain images in the testing phase. We follow this experiment setting over our experiments.

IV-D1 Effectiveness of Patch-based Output Space Adversarial Learning

the Drishti-GS and RIM-ONE-r3 datasets both provide the training and testing images splits. Therefore, for the Drishti-GS dataset, we used the REFUGE training dataset as the source domain and the training part of the Drishti-GS dataset as the target domain to train our pOSAL framework. We then report the segmentation performance of our method on the testing dataset of Drishti-GS. We conducted experiments with the same dataset setting for the RIM-ONE-r3 dataset.

Table II presents the segmentation results on the Drishti-GS and RIM-ONE-r3 testing datasets. For each dataset, we show the segmentation performance of our pOSAL framework (pOSAL) and the segmentation network only (pOSALseg-S) to demonstrate the effect of the proposed output space adversarial learning. We used the REFUGE training dataset to train the pOSALseg-S model and directly evaluated it on the Drishti-GS and RIM-ONE-r3 testing images. It is observed that the pOSAL consistently improves the DI of optic cup and disc and the $\delta$ on the Drishti-GS and RIM-ONE-r3 datasets compared with pOSALseg-S. On the RIM-ONE-r3 dataset, we achieve 4.3% and 8.6% DI improvement for the cup and disc segmentation with the patch-based output space adversarial learning, while we also achieve 2.2% and 2.1% DI improvement for OC and OD on the Drishti-GS dataset, respectively. Since the domain discrepancy between the REFUGE training data and RIM-ONE-r3 data is larger than the difference between REFUGE training data and Drishti-GS data (see Fig. 5), the absolute DI values of optic cup and disc on RIM-ONE-r3 is lower than those on Drishti-GS. These comparisons demonstrate the patch-based output space adversarial learning can alleviate the performance degradation among the datasets with domain shift.

IV-D2 Qualitative Results

we show some qualitative results of the optic OD and OC segmentation on the Drishti-GS dataset in Fig. 6. For the pOSALseg-S method without domain adaptation, it can locate the approximate location but fails to generate accurate boundaries of OD and OC due to the low image contrast at the boundary between OD and OC, as well as between OD and background (especially columns A, B and E in Fig. 6). In contrast, our proposed method successfully localizes the OD and OC and further preserves the shape prior and generates more accurate boundaries.

IV-D3 Comparison with other Segmentation Methods

we also report the segmentation performance of some supervised learning methods in literature for the above two datasets. In these methods, the networks were trained with the training split of the dataset in a supervised way and evaluated on the testing part of the related datasets. Besides the methods in the literature, we also trained our segmentation network with the training data and report the segmentation performance on the testing images (denoted as pOSALseg-T) to show the effectiveness of our designed segmentation network with the morphology-aware segmentation loss. We show these results in Table II. As we can see our segmentation network (pOSALseg-T) can produce better DI for the optic cup and disc segmentation compared with other supervised methods on both of the Drishti-GS and RIM-ONE-r3 datasets, showing the effectiveness of the segmentation network design. In another aspect, it is observed that the optic cup and disc segmentation performance of our pOSAL framework on the Drishti-GS dataset is very close to that of these supervised methods, which further indicates the effectiveness of the proposed patch-based output space adversarial learning.

IV-D4 Comparison with Different Domain Adaptation Approaches

as far as we know, we are not aware of any previous works that explores domain adaptation for optic disc and cup segmentation. Therefore, we compared our pOSAL framework with several unsupervised domain adaptation ideas in other medical image analysis and natural image processing tasks. Specifically, we compared our pOSAL framework with a Cycle-GAN based unsupervised domain adaptation method TD-GAN [32], a latent feature alignment method [48], and a recent domain adaptation method for eye vasculture segmentation [31]. To show the effectiveness of our patch-based discriminator, we also implemented a pixel-based discriminator for adversarial learning (denoted as OSAL-pixel). Table III presents the performance of different domain adaptation methods on the Drishti-GS and RIM-ONE-r3 datasets. All the methods adopted the same segmentation network architecture for a fair comparison. As we can see, our pOSAL framework achieves the best performance for the optic cup and disc segmentation among these unsupervised domain adaptation methods on the Drishti-GS and RIM-ONE-r3 datasets. In another aspect, the patch-based adversarial learning outperforms the pixel-level discriminator (OSAL-pixel) and the image-level discriminator (Javanmardi et al. [31]) for adversarial learning, as it considers the local and global context information simultaneously.

IV-D5 Performance of Glaucoma Screening

the vertical CDR value is one of the important indicators for glaucoma screening. Therefore, we provide the glaucoma diagnose performance based on our segmentation method. Specifically, we make use of the segmented OD and OC masks to calculate the vertical CDR value $p_{i}$ for the $i$th image. Then the normalized CDR values $\hat{p}_{i}$ of the $i$th image can be calculated using

[TABLE]

where $p_{max}$ and $p_{min}$ are the maximum and minimum vertical CDR values, respectively, through all the testing images. We report the Receiver Operating Characteristic (ROC) curve and Area Under ROC Curve (AUC) for glaucoma screening evaluation in Fig. 7.

IV-E Results of the REFUGE Challenge

We report the results for the optic disc and cup segmentation task of the REFUGE challenge in conjunction with MICCAI 2018. The challenge datasets consist of three parts: a training dataset, a validation dataset, and a testing dataset. The validation and testing datasets are acquired with the same cameras and the detailed information is shown in Table I. The testing images were evaluated via an on-site part, where the participants had four hours to acquire the testing images and submit the prediction results to avoid manually tuning the hyper-parameters. We treated the training images as the source domain and the validation images as the unlabeled target domain to train our pOSAL framework. The testing image prediction was then acquired by an ensemble of five models to improve the segmentation performance further. Other participant teams also utilized the ensemble scheme to generate the final testing prediction (e.g., team Masker).

There were 12 teams selected to participate in the onsite REFUGE challenge for the OD and OC segmentation task, and the challenge results are listed in Table IV (The leaderboard is in the challenge website222https://refuge.grand-challenge.org/Results-Onsite_TestSet/). Each team was only allowed for one submission, and the teams were ranked according to the following weighted sum of three metrics:

[TABLE]

where $R_{DI_{cup}}$, $R_{DI_{disc}}$, and $R_{\delta}$ denote the rank of $DI_{cup}$, $DI_{disc}$ and $\delta$ criteria, respectively. A lower $S_{f}$ suggests a better final rank. All these methods utilized deep neural networks for OD and OC segmentation. Some methods made use of other datasets (e.g., ORIGA [6] and IDRiD333https://idrid.grand-challenge.org) as the extra training data to improve the model generalization capability, while we only used the training and validation datasets provided by the organizer. In Table IV, it is observed that our pOSAL framework outperforms the second-ranking team Masker by around 1.4% on the optic disc DI, while we achieve compelling performance on both the DI of optic cup and CDR $\delta$. Overall, our pOSAL framework achieves the best overall ranking score $S_{f}$ and the first place in this challenging task, demonstrating the effectiveness of pOSAL. We also visually compare the appearance difference of the results with and without the output space adversarial learning using a single model. As shown in Fig. 8, our pOSAL framework could retain the elliptical features and push the optic cup within the optic disc to produce better visual results.

Besides, we further validate the effectiveness of the patch-based output space adversarial learning on the REFUGE validation dataset. Specifically, we randomly divided the 400 validation images into two equal-sized parts and used them as the unlabeled target domain training data to train the network and the target domain testing data to evaluate the network, respectively. We report the performance of our pOSAL framework and the same network without domain adaptation (pOSALseg-S) in Table V. As we can see, the presented pOSAL framework also improves the DI for optic cup and disc on the REFUGE validation dataset.

We compared the effect of different loss functions to the segmentation network. Specifically, we divided the 400 REFUGE training images into 320 and 80 images to train and evaluate the network, respectively, with different loss functions. The results are shown in Table VI. We can find that the Dice Loss achieved a better DI for OC and a comparable DI for OD compared with the Cross Entropy Loss. When combined with the smooth loss, the proposed morphology-aware segmentation loss achieves the best DI of OD and OC predictions, suggesting that the morphology-aware segmentation loss produces high-quality predictions.

We also provide the glaucoma screening evaluation results here for readers’ reference. We directly utilized the segmentation results of our pOSAL framework to calculate the vertical CDR values to diagnose glaucoma following the method on previous two datasets. Since we cannot access the ground truth of the glaucoma, we only report the AUC of glaucoma screening on the challenge testing dataset. The AUC value is $0.9644$, ranking third (Team CUHKMED) in the onsite challenge444https://refuge.grand-challenge.org/Results-Onsite_TestSet/.

V Discussion

The optic disc to cup ratio has been recognized as an essential attribute for glaucoma screening, so a high-quality automatic segmentation method is highly demanded in clinical practice. Although plenty of works worked on this problem, there still exists a gap between research works and clinical practice due to the lack of annotations, the noisy or sparse annotations of clinical applications, and the domain shift between training images and real testing images. In this work, we focus on developing unsupervised domain adaptation methods to enable optic disc and cup segmentation applied to clinical applications. The key insight of our method is to encourage the target domain predictions closer to the source ones, since the OD and OC geometry structure should be reserved for source and target domain images. The extensive experiments on three public fundus image datasets have sufficiently demonstrated the potential of our method in generalizing the segmentation network to unlabeled target domain images.

In our method, we first used an extraction network to crop an ROI image before performing the segmentation. To show the necessity of the ROI extraction, we conducted another experiment to see the overall performance of the extraction network $E$. We trained a new extraction network $E$ with two outputs instead of only the optic disc. The performance of OC and OD on the Drishti-GS and RIM-ONE-r3 datasets are shown in Table VIII. It is observed that the segmentation performance of using only the extraction network $E$ is much lower than the full method pOSAL. The results here verify the effectiveness of the two-stage pipeline. A good ROI is the basis of good segmentation results in the two-stage pipeline. In some cases, the boundary between the OD and background is unclear, so the OD may not in the center of the ROI. To avoid this kind of situation, the ROI size is needed to be designed properly. In our experiments, the width and height of ROI are about twice larger than that of the OD, which helps relax the location deviation. We found that all of the OD and OC regions are covered by the cropped ROI under this experiment setting.

Currently, numerous works are focusing on computation-efficient network design [14, 49] to promote the deep learning applications for mobile devices with limited computing power. In our work, we used a MobileNetV2 [14] as the network backbone to reduce the computation cost. We compared the segmentation performance, parameter numbers, and testing time cost of the original backbone: Xception [13] and MobileNetV2 [14] in Table VII. It is observed that the MobileNetV2 backbone has fewer parameters and can reduce the testing time by half with similar performance compared with the Xception backbone. This comparison indicates that we could develop more lightweight network architecture to promote the development of mobile applications for glaucoma screening.

Although our network can be generalized to unlabeled target domain images, collecting extra unlabeled images from the target domain is needed to train the network. Moreover, it is necessary to re-train a new network when the image comes from a new target domain. In practice, the unlabeled target domain images may not be available during the training stage. Therefore, in the future, we would explore the domain generalization techniques [50, 51, 52] to tackle this problem without the demand for many target images.

VI Conclusion

We presented a novel patch-based Output Space Adversarial Learning framework to segment optic disc and cup from different fundus images. We first employed a lightweight and efficient network with the morphology-aware segmentation loss to generate accurate and smooth predictions. To tackle the domain shift between the source and target domains, we exploited unsupervised domain adaptation model to improve the generalization of the segmentation network. Particularly, the patch-based output space adversarial learning was designed to capture the local statistics of the output space and guide the segmentation network generate similar outputs for the images from the target and source domains. We also performed extensive experiments on three public retinal fundus image datasets to demonstrate the significant improvements and the effectiveness of the presented pOSAL framework. More effort will be involved to extend this framework to other medical image analysis problems in the near future.