Closed-Loop Adaptation for Weakly-Supervised Semantic Segmentation

TL;DR

This paper introduces a closed-loop self-adaptation mechanism for weakly-supervised semantic segmentation, leveraging feedback chains to improve seed generation and region expansion, resulting in better performance and efficiency.

Contribution

It proposes a novel feedback-based framework that enhances existing methods by dynamically refining seeds and regions through two feedback chains.

Findings

Outperforms state-of-the-art methods on PASCAL VOC 2012

Reduces computational and memory requirements

Improves segmentation accuracy through feedback loops

Abstract

Weakly-supervised semantic segmentation aims to assign each pixel a semantic category under weak supervisions, such as image-level tags. Most of existing weakly-supervised semantic segmentation methods do not use any feedback from segmentation output and can be considered as open-loop systems. They are prone to accumulated errors because of the static seeds and the sensitive structure information. In this paper, we propose a generic self-adaptation mechanism for existing weakly-supervised semantic segmentation methods by introducing two feedback chains, thus constituting a closed-loop system. Specifically, the first chain iteratively produces dynamic seeds by incorporating cross-image structure information, whereas the second chain further expands seed regions by a customized random walk process to reconcile inner-image structure information characterized by superpixels. Experiments on PASCAL VOC 2012 suggest that our network outperforms state-of-the-art methods with significantly less computational and memory burden.

Tables

Table 1. Table 1 : Comparison of different weakly-supervised semantic segmentation methods on PASCAL VOC 2012 val set. The best two methods are marked with red and blue , respectively.

Method	bkg	plane	bike	bird	boat	bottle	bus	car	cat	chair	cow	table	dog	horse	motor	person	plant	sleep	sofa	train	tv	mIoU
EM-Adapt	66.3	44.4	17.9	35.1	30.0	34.5	47.0	46.1	45.1	18.8	37.6	26.9	42.0	44.1	42.1	38.9	27.7	37.0	23.7	49.3	43.0	38.0
STC	84.5	68.0	19.5	60.5	42.5	44.8	68.4	64.0	64.8	14.5	52.0	22.8	58.0	55.3	57.8	60.5	40.6	56.7	23.0	57.1	31.2	49.8
SEC	82.2	61.7	26.0	60.4	25.6	45.6	70.9	63.2	72.2	20.9	52.9	30.6	62.8	56.8	63.5	57.1	32.2	60.6	32.3	44.8	42.3	50.7
MCOF	85.8	74.1	23.6	66.4	36.6	62.0	75.5	68.5	78.2	18.8	64.6	29.6	72.5	61.6	63.1	55.5	37.7	65.8	32.4	68.4	39.9	56.2
DSRG	87.5	73.1	28.4	75.4	39.5	54.5	78.2	71.3	80.6	25.0	63.3	25.4	77.8	65.4	65.2	72.8	41.2	74.3	34.1	52.1	53.0	59.0
Ours	88.0	76.3	33.7	79.0	48.7	62.5	76.0	69.0	79.5	23.4	66.3	13.7	78.1	69.8	67.3	71.1	39.1	74.4	33.6	51.7	58.8	60.0

Table 2. Table 2 : Comparison of different weakly-supervised semantic segmentation methods on PASCAL VOC 2012 test set. The best two methods are marked with red and blue , respectively. The “-” denotes unknown values that were not reported by the corresponding paper.

Method	bkg	plane	bike	bird	boat	bottle	bus	car	cat	chair	cow	table	dog	horse	motor	person	plant	sleep	sofa	train	tv	mIoU
EM-Adapt	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	39.6
STC	85.2	62.7	21.1	58.0	31.4	55.0	68.8	63.9	63.7	14.2	57.6	28.3	63.0	59.8	67.6	61.7	42.9	61.0	23.2	52.4	33.1	51.2
SEC	83.5	56.4	28.5	64.1	23.6	46.5	70.6	58.5	71.3	23.2	54.0	28.0	68.1	62.1	70.0	55.0	38.4	58.0	39.9	38.4	48.3	51.7
MCOF	86.8	73.4	26.6	60.6	31.8	56.3	76.0	68.9	79.4	18.8	62.0	36.9	74.5	66.9	74.9	58.1	44.6	68.3	36.2	64.2	44.0	57.6
DSRG	87.9	69.5	32.1	74.2	33.7	59.4	74.9	71.5	80.1	21.9	66.8	32.7	76.4	72.5	76.6	73.4	49.9	73.8	43.4	42.0	55.2	60.4
Ours	88.0	73.9	31.3	75.2	44.9	61.0	77.8	69.6	78.9	24.9	71.9	20.2	74.6	75.0	75.3	74.3	47.1	76.2	45.0	41.1	56.3	61.1

Table 3. Table 3 : The number of parameters, the number of floating-point operations (FLOPS), and the mIoU values of our method, MCOF and DSRG.

Method	Params (M)	FLOPS (G)	mIoU
MCOF	about 40	about 201	57.6
DSRG	37.8	140.8	60.4
Ours	20.6	101.9	61.1

Table 4. Table 4 : Comparison of semantic segmentation method under different supervision types on PASCAL VOC 2012 set.

Method	Supervision types	val	test
FCN	fully-supervised	-	62.2
DeepLab	fully-supervised	67.6	70.3
WSSL	bounding-box	60.6	62.2
BoxSup	bounding-box	62.0	64.6
RAWK	scribble	61.4	-
ScribbleSup	scribble	63.1	-
What’sPoint	points	46.0	43.6
Ours	image-level tags	60.0	61.1

Table 5. Table 5 : Comparison of our method under different settings on PASCAL VOC 2012 val set. The “accu” denotes the pixel accuracy and the “fIoU” denotes frequency weighted IoU.

	accu	mIoU	fIoU
baseline	87.6	57.4	79.3
baseline + F1	88.0	57.8	80.0
baseline + F2	88.2	58.4	80.3
baseline + F1+ F2	89.3	60.0	81.4

Table 6. Table 6 : Performance of our method for different w 𝑤 w .

$w$	accu	mIoU	fIoU
0.10	88.8	59.1	80.7
0.20	89.3	60.0	81.4
0.30	89.2	59.4	81.4
0.40	89.0	58.7	80.9

Equations

$$S_{new}=(1-w)\times S_{old}+w\times N_{out},$$

$$M_{rel}=M_{siml}\odot M_{adj},$$

$$\hat{S}=\mathrm{T}{\mathrm{h}}_1(S)\times M_{rel}\odot \mathrm{T}{\mathrm{h}}_2(N_{out}),$$

$$\hat{S}=\mathrm{T}{\mathrm{h}}_1(S)\times M_{rel}\odot \mathrm{T}{\mathrm{h}}_2(N_{out})\times M_{rel}\odot \mathrm{T}{\mathrm{h}}_2(N_{out}).$$

Full text

††journal: Journal Name

1 Introduction

Semantic segmentation of an image refers to the task of assigning each pixel a categorical label, e.g., motorcycle or person [30]. Owing to the rapid development of deep learning, tremendous progress has been made for fully-annotated images. Notable examples include the DeepLab [3] and the PSPNet [30]. These methods assume that the pixel-level labels are available immediately upon request. However, this assumption is over-optimistic, since it could involve the annotation of data by expensive means in terms of cost and labor time. As a result, weakly-supervised semantic segmentation which only requires a few weak labels, such as bounding box [4], scribble [17], points [1] and tags [18], has attracted increasing attention.

This paper focuses on weakly-supervised semantic segmentation with image-level tags. Under the circumstance, the lack of structure information on the organization of image pixels prevents network learning from labels directly, which makes the problem of weakly-supervised semantic segmentation ill-conditioned. Consequently, how to recover the cross-image structure information and the inner-image structure information becomes a pivotal issue. The cross-image structure information describes how to organize pixels for a specific semantic category across multiple images. For example, given this kind of information for the person category, we can cluster pixels (in multiple images) into local regions (e.g., heads and faces) that are discriminative to recognize a person. Unlike cross-image structure information, the inner-image structure information tells us how to organize pixels (in a single image) based on their low-level features like textures or colors. This kind of information always contains details about object boundaries.

In this paper, we propose Dual-Feedback Network (DFN), a closed-loop system with two feedback chains, for weakly-supervised semantic segmentation. The architecture of DFN is shown in Figure 3. Our general idea is to adopt a divide-and-conquer strategy to recover both cross-image and inner-image structure information, and each feedback chain is imposed to compensate for the lack of one type of structure information. The first chain updates the labels progressively to correct errors made by static pseudo-label (see Figure 3(c)). Specifically, we generate initial seeds using Class Activation Mapping (CAM) [31] and set it the pseudo-label to train the network in the first few epochs. After that, the network output is routed back to the initial seeds to amend their probability distributions. The first feedback chain repeats this procedure several iterations until seeds are not changed, whereas the second feedback chain uses a customized random walk to integrate inner-image structure information characterized by a relationship matrix built upon superpixels into network training (see Figure 3(d)). Our method demonstrates state-of-the-art performance on the PASCAL VOC 2012 segmentation dataset.

To summarize, our contributions are threefold:

We interpret weakly-supervised semantic segmentation as a closed-loop problem, and introduce two feedback chains to recover, respectively, the cross-image structure information and the inner-image structure information. We also demonstrate that the new network, i.e., DFN, can be trained end-to-end. 2. 2.

We construct a relationship matrix based on superpixels to recover inner-image structure information. This method is computational and memory efficient, and robust to noisy labels. 3. 3.

Our method outperforms existing weakly-supervised semantic segmentation methods with image-level tags. Especially, the mean Intersection-Over-Union (mIOU) values of our method are 60.0% and 61.1% on val and test sets, respectively.

2 Related Work

Compared with fully-supervised semantic segmentation, the main issue for weakly-supervised semantic segmentation is the lack of structure information that tells us how to organize the pixels. This problem becomes more serious if only image-level tags are available. Existing weakly-supervised semantic segmentation with image-level tags can be divided into two categories.

The first category uses a specialized loss function or objective to recover the structure information. [21] initially suggests a multi-instance loss (MIL) function to recognize the most discriminatory region for each category. Later, [22] adds some linear constrains into an iteratively updating process to restrict the structure information of the output, leading to an improved segmentation result. On the other hand, [6] proposes a weakly-supervised learning transfer layer to discover complementary regions for the subcategories which belong to the same category, whereas [16] builds a guided attention inference network and uses an attention mining loss to constrain the output. However, this kind of methods only restricts a small part of pixels in the network output.

Different from the first category, the methods in the second category generate pseudo-label to incorporate the structure information, and use the new generated labels to constrain all of the pixels in the network output. For example, [14] obtains the pseudo-label using CAM, [23] uses the activations from the middle layers of VGG-16 [24] to generate some candidate regions, whereas [11] sets attention map of each image as labels. [29] proposes a simple-to-complex mechanism to generate the pseudo-label iteratively. The authors then apply an erasing mechanism into their network structure and discover new regions to expand the truth area in the pseudo-label [28]. Despite substantial progress made by these methods, they always use static pseudo-label which is likely to accumulate errors incurred by initial inaccurate seeds or a dense conditional random field (DenseCRF) model [15] that is sensitive to noisy labels.

In this paper, we introduce two feedback chains into existing weakly-supervised semantic segmentation network to form a close-loop system, which enables iterative extraction of robust structure information and dynamical correction of errors made by inaccurate seed localization. Specifically, the first feedback chain updates the seed by the network output to recover the cross-image structure, whereas the second feedback chain incorporates robust inner-image structure information characterized by a relationship matrix built upon superpixels to further expand seed regions. It is worth noting that deep seed region growing (DSRG) [12] also uses dynamic seed. However, it does not correct the inaccurate seed localization explicitly, which constrains the upper bound of segmentation performance. On the other hand, Mining Common Object Features (MCOF) [27] also suggests using superpixels to robustly rebuild inner-image structure information in the presence of noisy labels. Unlike MCOF, we use a simple relationship matrix to characterize superpixels, rather than an extra network that requires significantly higher computational and memory burden. Moreover, experiments demonstrate that our network outperforms both DSRG and MCOF with a large margin.

3 Our Approach

In this section, we elaborate our proposed DFN, especially the way to recover robust inner-image structure information with the aid of relationship matrix.

3.1 Overview

Similar to [14, 27, 12], we use DeepLab-LargeFOV network [3] as the baseline network and build two feedback chains as shown in Figure 4. Specifically, we first generate initial seeds using CAM and the saliency detection method in [13]. Then, in the second feedback chain, we integrate the robust inner-image structure information characterized by a relationship matrix built upon superpixels to produce the mixed seeds. Next, the baseline network is trained with the obtained mixed seeds under the seed loss function [12] and the boundary loss function [14]. Using the network output, we update the seeds progressively along the first feedback chain. We repeat above procedures until there is no change on seeds.

3.2 The First Feedback Chain

Although the seed cues localized by CAM or saliency maps are discriminative and precise, they are sparsely distributed and still contain tiny errors. To correct these errors, we adopt an updating mechanism to adjust seeds progressively. Specifically, in each iteration, we update the seed locations with the network output using the following simple formula:

[TABLE]

where $S_{new}$ denotes seeds in the current iteration, $S_{old}$ denotes seeds from the last iteration, $N_{out}$ is the network output, and $w\in[0,1]$ is a weighting factor that balances the update rate.

One should note that there are two key points behind Eq. (1). First, to preserve the boundary information and reduce the memory cost, the seeds are stored based on superpixels, rather than 41$\times$ 41 pixel blocks as in [29] and [12]. Second, in the interval of two subsequent seeds iterations, the network is trained independently with $3$ epochs. Moreover, we monitor the seeds states in different iterations to determine whether to continue the update. Specifically, each superpixel in seeds is characterized by a vector with its element corresponds to the probability to each category. Then, for one superpixel, we view it unchanged if the sum of absolute differences between elements of two vectors in two subsequent iterations is less than $0.1$. We stop seeds update if $95\%$ of superpixels are unchanged. Figure 5 shows the evolution of seeds at different epochs. As can be seen, with the increase of the number of iterations, initial errors are corrected and the dynamic seeds get closer to the ground truth.

3.3 The Second Feedback Chain

3.3.1 Robust Inner-Image Structure Information Recovery

Previous works use DenseCRF to build the relationships among pixels to recover the inner-image structure information. Unfortunately, the DenseCRF relies heavily on the color and location of each pixel, which makes it sensitive to noisy pixels. To address this limitation, we recover inner-image structrue information based on superpixels generated by Felzenszwalb’s method [8] and refined by Region Adjacency Graph [25] (see the first row of Figure 7 as an example).

The construction of relationship matrix consists of four steps. First, we generate Euclidean distance matrix for superpixels using zoom-out features [19]. However, unlike [19], we only use the first two convolutional layers in VGG-16 to obtain low-level features. Nevertheless, owing to the diversity of objects under complex backgrounds in each image, the Euclidean distance with absolute values cannot precisely reflect the similarity between any two superpixels. Therefore, we then transform the Euclidean matrix to a simple similarity matrix (denoted $M_{siml}$), as shown in Figure 7(e). Specifically, we pick the $10$ smallest values in each row of the Euclidean matrix and set their values to $1$, indicating that they are mutually similar. We then set the values of remaining elements in each row of the Euclidean matrix to [math], suggesting that they are not similar. Next, we construct an adjacency matrix (denoted $M_{adj}$) for superpixels to incorporate their location information. If two superpixels are neighboring, the corresponding element in the adjacency matrix is $1$, otherwise [math] (see Figure 7(f)). We finally obtain a relationship matrix (denoted $M_{rel}$) with the following formula (see Figure 7(f)):

[TABLE]

where $\odot$ denotes entry-wise product.

3.3.2 Customized Random Walk

After recovering the inner-image structure information characterized by a relationship matrix built upon superpixels, the second feedback chain performs a customized random walk to incorporate both the inner-image structure information and the cross-image structure information into network training.

The input to our customized random walk includes relationship matrix, dynamic seeds, and network output. However, a preprocessing to dynamic seeds and network output is required herein. The reason is that each superpixel in seeds or network output is characterized by a vector with its element corresponds to the probability to each category. It means that if the value of the maximum element in the vector (of a superpixel) is small, we have a high classification uncertainty to assign this superpixel to its corresponding category. Moreover, the superpixels with high classification uncertainty will mislead the direction of seed expansion in random walk. Therefore, it is necessary to filter out these superpixels at first. To this end, we perform image thresholding to both dynamic seeds and network output. Specifically, we use a threshold function (denoted ${\rm Th_{1}}(\cdot)$) coupled with two hyperparameters $\alpha_{fg}$ and $\alpha_{bg}$ to distinguish foreground and background in seeds. Similarly, we use another threshold function (denoted ${\rm Th_{2}}(\cdot)$) coupled with hyperparameters $\beta_{fg}$ and $\beta_{bg}$ to distinguish foreground and background in the network output. For example, if the maximum element of one superpixel belong to the foreground in dynamic seeds is less than $\alpha_{fg}$, we drop out this superpixel in the following customized random walk.

By using the processed seeds as the initial state and the relationship matrix as the transition probability matrix, the basic random walk [2] generates candidate segmentation regions with “${\rm Th_{1}}(S)\times M_{rel}$”, where “$\times$” denotes matrix multiplication. However, to remove false segmentation, we use the network output $N_{out}$ to guide the refinement of result from the basic random walk process. Therefore, the result $\hat{S}$ of our one-step customized random walk is given by:

[TABLE]

where $\odot$ denotes entry-wise product. We repeat the operation of “$\times M_{rel}\odot{\rm Th_{2}}(N_{out})$” $n$ times on the initial state ${\rm Th_{1}}(S)$ to obtain the $n$-step random walk result. For example, the result of two-step customized random walk is given by:

[TABLE]

We present two examples of our customized random walk in Figure 10. By merging the robust inner-image structure information into the dynamic seeds, the obtained mixed seeds fit well with object boundaries and expand confident regions.

4 Experiments

4.1 Experiment Setup

Dataset and Evaluation Metrics We evaluate the proposed DFN on the PASCAL VOC 2012 segmentation benchmark dataset [7] which constains 20 foreground object classes and one background class. The segmentation part of VOC 2012 dataset is split into three parts: training (train, 1464 images), validation (val, 1449 images) and testing (test, 1456 images). Same to [12] and [27], the training set is extended with additional images from [10], resulting in an augmented set of 10582 images. Following the common practice, we use the mean Intersection-Over-Union (mIoU) criterion to compare our method with other approaches on both val and test sets. We report our results on standard val set with the ground truth segmentation masks are available. For the test set, we submit the results of our final best model to the official evaluation server.

Implementation details For all the experiments, we use the DeepLab-LargeFOV network [3] as the baseline segmentation architecture which is initialized from VGG-16 [24] pre-trained on ImageNet [5]. We use a mini-batch of $128$ images for SGD and initial learning rate of $3e$-$3$ which is decreased by a factor of $3$ every $5$ epochs. The momentum is $0.9$, the dropout rate is $0.5$ and the total epochs of training is $20$.

For the first feedback chain, we set $w$ to $0.2$ and start to update the seeds every $3$ epochs after the $10$-th epoch. For the second feedback chain, we set $\alpha_{fg}$ to $0.90$, $\alpha_{bg}$ to $0.90$, $\beta_{fg}$ to $0.75$, and $\beta_{bg}$ to $0.90$. In addition, we perform two steps of customized random walk. In the test phase, same to [14, 12], the fully-connected CRF [15] and the multi-scale prediction [3] are applied with their default parameters.

Reproducibility We use tensorflow to implement our approach. The code will be released soon. Anyone can use it and trains on a single NVIDIA GTX 1080TI GPU for about $15$ hours.

4.2 Comparison with Previous Methods

Table 1 and Table 2 report the mIoU values of our method on PASCAL VOC 2012 val and test sets, respectively, against previous state-of-the-art methods, namely EM-Adapt [20], STC [29], SEC [14], MCOF [27], and DSRG [12]. As can be seen, our method outperforms all compared methods in terms of mIoU value under the same experimental setting. Although MCOF [27] uses an extra network to recover structure information with superpixels, our method achieves a performance gain of $3.8\%$ and $3.5\%$ on val and test sets, respectively. Meanwhile, the improvement is $1.0\%$ and $0.7\%$ respectively compared to DSRG [12].

Computational and Memory Cost We summarize the number of parameters and the number of floating-point operations (FLOPS) in Table 3. Compared with DSRG and MOCF, our network only requires approximately $50\%$ of parameters and reduces at least $30\%$ FLOPS. This result suggests that our method is more computational and memory efficient.

Different Supervision Types We compare our network with other methods under different types of supervisions. They are FCN [18], DeepLab [3], WSSL [20], BoxSup [4], RAWK [26], ScribbleSup [17], and What’sPoint [1]. As can be seen in Table 4, our network achieves comparable performance to other methods that require stronger supervisions, e.g., the WSSL, the RWAK, or even the fully-supervised FCN. This result also suggests that there is a large performance gap between fully-supervised semantic segmentation and the weakly supervised semantic segmentation, especially for the points supervisions.

Quantitative Results The segmentation results shown in Figure 11 corroborate our quantitative evaluations. Note that, our method can generate precise segmentation results even for images containing complex backgrounds. However, our method is likely to fail when there are multiple small and dense objects on top of another larger object. Taking the last row of Figure 11 as an example. Our method misclassified most of pixels in the table into background. One possible reason is that there are lots of plates and foods on the table, such that various of colors and textures of these small and dense objects make our method hard to generate a precise relationship matrix.

4.3 Ablation Studies

To validate the effects of different components, we perform some ablation experiments under different settings. In Table 5, we summarize the performance of our network in different degrading settings. Specifically, the “baseline” indicates our baseline network without two feedback chains, the “baseline+F1” indicates only integrating the first feedback chain into the baseline network, the “baseline+F2” indicates only integrating the second feedback chain into the baseline network, whereas the “baseline+F1+F2” indicates integrating both two chains. We observe $0.4$ percent gain in mIoU using the first feedback chain. This suggests that our dynamic seeds update mechanism indeed reduces more errors than DSRG. Moreover, by comparing the mIoU value of “baseline+F2” with that of MCOF ($56.2$), although both methods attempt to recover inner-image structure information based on superpixels, our relationship matrix achieves $2.2$ percent performance gain, with significantly less computational and memory burden.

4.4 Effects of Hyperparameters

We finally evaluate the performance of our network with respect to different hyperparameter settings and specify the way to pinpoint their values. The hyperparameter $w$ in Eq. (1) represents the update rate in the first feedback chain. A large $w$ makes the dynamic seeds cannot supervise network training effectively, whereas a small $w$ slows down the correction of errors incurred by initial inaccurate seeds. We go through possible values of $w$ in a large range and select the best-performing one. From Table 6, the optimal value of $w$ is $0.20$.

In the second feedback chain, there are four hyperparameters, i.e., $\alpha_{fg}$, $\alpha_{bg}$, $\beta_{fg}$ and $\beta_{bg}$. We set their values to $\alpha_{fg}=0.90$, $\alpha_{bg}=0.90$, $\beta_{fg}=0.75$ and $\beta_{bg}=0.90$. These hyperparameters are tuned with a coarse-to-fine procedure. In the coarse module, we roughly determine a satisfactory range for each hyperparameter (for example, the range of $\alpha_{fg}$ is $[0.6,1.0]$). Then, in the fine-tuning module, we divide these parameters into two groups based on their correlations: (1) $\{\alpha_{fg},\alpha_{bg}\}$; (2) $\{\beta_{fg},\beta_{bg}\}$. When we test values of one group of hyperparameters, another group is set to default values, i.e., the mean value of the optimal range given by the coarse module. In each group, the hyperparameters are tuned with grid search (for example, $\alpha_{fg}$ is tuned at the range $[0.6,1.0]$ with an interval $0.1$). We finally pinpointed the specific value as the one that can achieve the highest mIoU value (for example, the final value of $\alpha_{fg}$ is $0.90$). We also test the sensitivity of these hyperparameters. To this end, we evaluate the performance variation for one hyperparameter with other three fixed. The results are shown in Figure 12. We can observe that in a wide range, our network outperforms the baseline with a large margin. It means that our performance is not sensitive to these four hyperparameters. Moreover, the performance variation for $\alpha_{fg}$ and $\alpha_{bg}$ seems less than that for $\beta_{fg}$ and $\beta_{bg}$. One possible reason is that seeds update slowly than network output.

5 Conclusion and future work

In this paper, we designed a closed-loop network architecture, namely Dual-Feedback Network, for weakly-supervised semantic segmentation with image-level tags by introducing two feedback chains. The proposed network can correct the errors made by initial inaccurate seed localization with a seed updating mechanism and increase the robustness to noisy labels by incorporating inner-image structure information from superpixels. Experiments on PASCAL VOC 2012 show that our network outperforms previous state-of-the-art methods under the same experimental condition. Our method is also much more computational and memory efficient. Moreover, it is robust to different parameter settings.

In the future, we will continue improving the performance of DFN. This is because our prediction results in some examples still suffer from non-smooth boundaries or inconsistent small holes. At the same time, we are also interested in applying generative adversarial nets [9] to recover the missing structure information.