Multi-stage Multi-recursive-input Fully Convolutional Networks for Neuronal Boundary Detection

TL;DR

This paper introduces a multi-stage, multi-recursive-input fully convolutional network for neuronal boundary detection in electron microscopy images, effectively handling multi-scale features and ambiguous boundaries to improve segmentation accuracy.

Contribution

It proposes a novel multi-stage network with recursive multi-scale inputs, inspired by biological visual systems, trained end-to-end for improved neuronal boundary detection.

Findings

Achieves promising results on EM segmentation datasets.

Effectively detects both thin and thick neuronal membranes.

Outperforms existing methods in boundary detection accuracy.

Abstract

In the field of connectomics, neuroscientists seek to identify cortical connectivity comprehensively. Neuronal boundary detection from the Electron Microscopy (EM) images is often done to assist the automatic reconstruction of neuronal circuit. But the segmentation of EM images is a challenging problem, as it requires the detector to be able to detect both filament-like thin and blob-like thick membrane, while suppressing the ambiguous intracellular structure. In this paper, we propose multi-stage multi-recursive-input fully convolutional networks to address this problem. The multiple recursive inputs for one stage, i.e., the multiple side outputs with different receptive field sizes learned from the lower stage, provide multi-scale contextual boundary information for the consecutive learning. This design is biologically-plausible, as it likes a human visual system to compare different possible segmentation solutions to address the ambiguous boundary issue. Our multi-stage networks are trained end-to-end. It achieves promising results on two public available EM segmentation datasets, the mouse piriform cortex dataset and the ISBI 2012 EM dataset.

Figures

Tables

Table 1. Table 1: Performance of alternative designs for network architectures and training strategies.

Alternative Design	$V_{Fscore}^{Rand}$
pre-trained single stage, 5-level (baseline)	0.9680
AD_I: 1-stage, 5-level	0.9688
AD_II: 2-stage, 4-level, end-to-end,	0.9739
multi-recursive-input
AD III: 2-stage, 5-level, end-to-end,	0.9410
single-recursive-input (level 5)
AD_IV: 2-stage, 5-level, end-to-end,	0.9656
single-recursive-input (level 4)
AD_V: 2-stage, 5-level, stepwise,	0.9762
multi-recursive-input
AD_VI: 2-stage, 5-level, end-to-end,	0.9819
multi-recursive-input
AD_VII: 3-stage, 5-level, end-to-end,	0.9866
multi-recursive-input

Table 2. Table 2: Neuronal boundary detection performance comparison between different methods. The values of V m ​ e ​ r ​ g ​ e R ​ a ​ n ​ d superscript subscript 𝑉 𝑚 𝑒 𝑟 𝑔 𝑒 𝑅 𝑎 𝑛 𝑑 V_{merge}^{Rand} and V s ​ p ​ l ​ i ​ t R ​ a ​ n ​ d superscript subscript 𝑉 𝑠 𝑝 𝑙 𝑖 𝑡 𝑅 𝑎 𝑛 𝑑 V_{split}^{Rand} correspond to the best Rand F-score V F ​ s ​ c ​ o ​ r ​ e R ​ a ​ n ​ d superscript subscript 𝑉 𝐹 𝑠 𝑐 𝑜 𝑟 𝑒 𝑅 𝑎 𝑛 𝑑 V_{Fscore}^{Rand} .

Method	$V_{merge}^{Rand}$	$V_{split}^{Rand}$	$V_{Fscore}^{Rand}$
N4 [8]	0.9619	0.9010	0.9304
VD2D [21]	0.9771	0.9174	0.9463
VD2D3D [21]	0.9891	0.9555	0.9720
${\text{M}}^2$FCN (1 stage)	0.9576	0.9802	0.9688
${\text{M}}^2$FCN (2 stage)	0.9759	0.9880	0.9819
${\text{M}}^2$FCN (3 stage)	0.9917	0.9815	0.9866

Table 3. Table 3: Comparison to published entries on the ISBI 2012 EM dataset  [ 29 ] . For full ranking of all submitted methods, please refer to the challenge website: http://brainiac2.mit.edu/isbi_challenge/leaders-board-new .

Method	$V_{Fscore}^{Rand}$
PolyMtl [10]	0.9806
${\text{M}}^2$FCN (ours)	0.9780
FusionNet [28]	0.9780
IAL IC [2]	0.9773
CUMedVision [5]	0.9768
FCN+LSTM [6]	0.9754
Unet [29]	0.9727

Equations

$$X^{(m)}=X\oplus S^{m-1,1}\oplus \dots \oplus S^{m-1,n}$$

$${\ell }^{m,n}({\mathbf{W}}^m,{\mathbf{w}}^{m,n};X^{(m)},Y)=$$

$$-\beta \sum _{j\in |B|}\log (1-\sigma (s_j^{m,n}))$$

$$-(1-\beta )\sum _{j\in |\bar{B}|}\log (\sigma (s_j^{m,n})).$$

$${\mathcal{L}}_s(\mathbf{W},\mathbf{w};X,Y)=$$

$$\sum _{m=1}^M\sum _{n=1}^N{\alpha }_{m,n}{\ell }^{m,n}({\mathbf{W}}^m,{\mathbf{w}}^{m,n};X^{(m)},Y),$$

$$S^{f,m}=\sum _{n=1}^N{h}_{m,n}{S}^{m,n},$$

$${\ell }^{f,m}({\mathbf{W}}^m,{\mathbf{w}}^m;X^{(m)},Y)=$$

$$-\beta \sum _{j\in |B|}\log (1-\sigma (s_j^{f,m}))-(1-\beta )\sum _{j\in |\bar{B}|}\log (\sigma (s_j^{f,m})).$$

$${\mathcal{L}}_f(\mathbf{W},\mathbf{w},\mathbf{h};X,Y)=$$

$$\sum _{m=1}^M{\alpha }_{f,m}{\ell }^{f,m}({\mathbf{W}}^m,{\mathbf{w}}^m;X^{(m)},Y),$$

$$(\mathbf{W},\mathbf{w},\mathbf{h}{)}^{\ast }=\arg \min ({\mathcal{L}}_s(\mathbf{W},\mathbf{w};X,Y)$$

$$+{\mathcal{L}}_f(\mathbf{W},\mathbf{w},\mathbf{h};X,Y)).$$

$$(S^{m,n},n=1\dots ,N)=F_m(X^{(m)},$$

$$({\mathbf{W}}^1{)}^{\ast },\dots ,({\mathbf{W}}^m{)}^{\ast },({\mathbf{w}}^{m,1}{)}^{\ast },\dots ,({\mathbf{w}}^{m,N}{)}^{\ast }).$$

$$S^{f,m}=\sum _{n=1}^N{h}_{m,n}^{\ast }{S}^{m,n}.$$

$$V_{Fscore}^{Rand}=\frac{2V_{merge}^{Rand}{V}_{split}^{Rand}}{V_{merge}^{Rand}+V_{split}^{Rand}},$$

$$V_{merge}^{Rand}=\frac{{\sum }_{ij}{n}_{ij}^2}{{\sum }_i({\sum }_j{n}_{ij}{)}^2},V_{split}^{Rand}=\frac{{\sum }_{ij}{n}_{ij}^2}{{\sum }_j({\sum }_i{n}_{ij}{)}^2},$$

Full text

1 Introduction

A central theme of neuroscience is to understand how a brain’s functions are related to its neuronal structure [23]. This is difficult because of the small size of neurons and the extremely high packing density of the neuropil, e.g., neurons densely packed axons and dendrites [13]. Recent advances in high-throughput serial section electron microscopy (EM) have made possible the imaging of large volumes of neuronal tissue at high resolution, allowing neuroscience experts to reconstruct neuronal circuits and study the interconnections of neurons [38]. However, analysis of a large number of EM images by expert annotators is laborious and even impractical [13], which drives the demand for efficient automated neuronal circuit reconstruction approaches.

Serial section EM produces a stack of 2D images by cutting sections of brain tissue. Due to the anisotropic resolutions of in-plane and out-of-plane, most neuronal circuit reconstruction approaches follow the following pipeline: (1) neuronal boundary detection on each 2D image, (2) neuronal structure segmentation based on the 2D boundary map, and (3) linking the neuronal segments across 2D images into a 3D reconstruction result.

This paper focuses on neuronal boundary detection on 2D images from serial section EM, also called membrane detection, which is the essential first step of the reconstruction pipeline. The challenges of this problem, as can be seen from Fig. 1, mainly lie in the following aspects: (1) The variation of the membranal thickness is large, as thin as a filament to as thick as a blob. (2) The noise of EM acquisition makes the membrane contrast to be low, inducing some membranal boundaries are even invisible. (3) The presence of confounding structures, such as mitochondria and vesicles, also increases the difficulty in membrane detection.

Driven by the rapid process of deep neural networks, more and more deep learning based methods have been proposed for neuronal boundary detection on EM images, achieving considerable progress [8, 5, 11, 28, 21]. However, most of them only focus on how to improve detection performance by using the deep learning strategies shown to be effective on general computer vision problems, like image classification [17, 37, 12], semantic segmentation [25] and boundary/symmetry detection [35, 42, 36]. For example, using fully convolutional networks enables holistic image training instead of patch-by-patch training [5]; deeply supervised learning hierarchical representations [5]; using residual structures rather than plain structures [11, 28]. Although these strategies indeed improve membrane detection results, they lack an interpretation for this problem, i.e., how can membranes be detected and intracellular structures be suppressed meanwhile, even the contrasts of intracellular structures to context are much higher than those of membranes?

In this paper, we propose multi-stage multi-recursive-input fully convolutional networks ($\text{M}^{2}$FCN) for neuronal boundary detection. The architecture of $\text{M}^{2}$FCN is shown in Fig. 2. The whole net consists of multiple stages, where each stage generates multiple side outputs by imposing supervision at different levels [20, 42] of the sub-net in this stage, and all these side outputs are concatenated with the original image to serve as the inputs for the next stage.

From a neurobiological prospective, this network architecture is biologically-plausible. First, as explained in [21], this recursive framework is in accord with the interplay process, named “countercurrent disambiguating process” [7], between the primary and higher visual cortical areas (V1 and V4, respectively) in monkeys’ brains, found by examining monkeys’ performances on contour detection tasks. The latter stages of our networks act as V4 to detect the overall “contour” of neuronal boundaries and feed top-down influence to the early stages, which acts as V1, to enhance the activation on neuronal boundaries while suppressing those on intracellular structures. Second, as pointed out in [13], the ambiguous neuronal boundaries make segmentation difficult, and this issue can only be resolved when explicitly comparing the different possible segmentation solutions, which is what the human visual system may compute when inspecting this situation. We can roughly think that of the multiple recursive inputs, computed at the different levels in the previous stage, provide different possible segmentation solutions for training of the next stage. We show that using multiple recursive inputs is very important in our experiments, as it leads to much better results than using a single recursive input.

From the deep learning view, our networks take several advantages of the latest deep learning strategies to facilitate the learning of a neuronal boundary detection model, including (1) holistic image training and prediction benefited from using the architecture of fully convolutional neural networks, (2) supervised multi-scale feature learning at each level of each stage and (3) learning multi-stages in an end-to-end fashion, different from previous recursive approaches [39, 21, 14, 9] which learn a series of classifiers stepwise. Using multiple recursive inputs rather than one is a major difference between our networks and [21]. Note that, the multiple recursive inputs for one stage are the multiple side outputs computed at the levels having different scales (receptive field sizes). In general, a side output with a small scale has better ability than one with larger scale for detecting a thin neuronal boundary between cluttered neurons. Conversely, a side output computed at a large scale can suppresses the false predictions on intracellular structures by using more image context. Therefore, these multiple recursive inputs provide richer information than one for the learning of next stage. We show that introducing these strategies in our networks can improve the performance of neuronal boundary detection.

We verify our networks on two public available EM segmentation datasets. One is the mouse piroform cortex EM dataset [21], a sizable and important EM dataset, contains 4 stacks of EM images of mouse piriform cortex, covering 460 images for training and 168 images for testing. We analyze alternative designs in our network architecture on this dataset and show it outperforms the state-of-the-arts [21]. The other is the dataset used for ISBI 2012 EM segmentation challenge [29], covering 30 images for training and 30 images for testing. Our networks can achieve a comparable result to the state-of-the art [28, 10] on this dataset.

In summary, our main contributions includes (1) end-to-end multi-stage networks architecture, in which each stage generates multiple side outputs by imposing supervision on different levels and feeds them into the next stage as the multiple recursive inputs. (2) using multiple recursive inputs rather than single recursive inputs can not only boost the performance of neuronal boundary detection, but also be biologically-plausible. (3) our networks achieve promising results on two public available EM segmentation datasets.

2 Related Work

Segmenting EM images of neural tissue is an important step to understand the circuit structure and the function of the brain [21, 28]. Early work of this topic needs to impose experts’ knowledge. For example, users need to label intracellular regions to allow graph cut segmentation, and also correct segmentation errors afterwards [41]. To reduce the amount of human labor required [13], automatic neuron segmentation became an active research direction, which follows the pipeline that detects neuronal boundaries by machine learning algorithms [19, 15, 18, 31] and then applies post-processing algorithms, such as watershed [26, 40, 43], hierarchical clustering [27, 24] and graph cut [16] algorithms, to boundary maps to obtain neuron segments. But early methods, which are based on hand-crafted features, tend to fail when the membrane is ambiguous.

With the rapid development of deep networks, segmentation and classification on medical images have undergone a vast revolution, from analyzing specific features of neuronal boundaries to designing fully automatic algorithm without experts’ knowledge. Recent deep learning based neuron segmentation methods are well suited to deal with ambiguous neuronal boundaries in EM images. One of the earliest works made by Ciresan et al. [8] used a succession of max-pooling convolutional networks as a pixel classifier, which estimated the probability of a pixel being a membrane. This method won the ISBI 2012 EM segmentation challenge [1]. Ronneberger et al. [29] presented a U-net structure with contracting paths, which captures multi-contextual information. Compared with the work in [8], the U-net replaces pooling operations by upsampling operators, which propagates context information from multiple feature channels to higher resolution layers. Fully convolutional networks (FCNs) [25] led to a breakthrough on semantic segmentation. With deep supervision and side outputs on FCN, a holistically-nested edge detector (HED) [42] was then proposed to solve the ambiguity in edge and object boundary detection. Due to the great success of these methods, Chen et al. [5] presented a deeply supervised contextual network to fuse the multi-level side-outputs to segment the membrane. Deep residual network (ResNet) [12] addresses the degradation (of training accuracy) problem by optimizing the residual mapping, which ranked the 1st on ILSVRC 2015 image classification challenge [30]. This inspired fully residual convolutional neural networks with nested short and long skip connections for membrane segmentation, proposed by Quan et al. [28] and Fakhry et al. [11]. Most of the recent deep learning based methods were motivated by the novel network architectures proposed in the computer vision community. Although these methods achieve improvements on membrane detection, they lack interpretation and comprehensive analysis. Our proposed multi-stage multi-recursive-input fully convolutional networks ($\text{M}^{2}$FCN) not only considers the advantages of the latest deep network architectures but also is biologically plausible.

The recursive training framework has been applied to many computer vision tasks, such as image labeling [39], instance segmentation [22], human pose estimation [34], and face alignment [9, 3]. However, they trained the recursive framework stepwise and only used one single recursive input. There are two image segmentation methods [32, 33] also fed multi-recursive inputs into next stage in the recursive training framework. But, their strategies to generate the multi-recursive inputs are different from ours. In the first one [32], the multi-recursive inputs for one stage were obtained by applying a series of Gaussian filters to the single output of the previous stage, but ours are the multiple outputs supervised at different levels in a deep network. The second one [33] downsampled an original input image into multiple input images with different resolutions and obtained the multi-recursive inputs from these multiple input images, but ours are computed from the same input image by using the hierarchy of a deep net. In addition, the multiple stages in their methods were trained in a stepwise manner. On the contrary, we embed the recursive learning in a deep network and first learn it in an end-to-end fashion.

Our work is related to the recursively trained network proposed in [21], which trains a Very Deep 2D (VD2D) network first, then a Very Deep 2D-3D (VD2D3D) network initialized with learned 2D representations from VD2D network is trained to generate the boundary map. There are two important differences between the networks in [21] and our method. (1). We use multiple recursive inputs with different receptive field sizes to incorporate multi-level contextual boundary information learned from the previous stage, while VD2D3D only uses the single output of VD2D as its recursive input. (2). We train our network in an end-to-end fashion to co-enhance the learning ability (e.g., detect membranes while suppress intracellular structure) of all stages, while [21] sequentially learns deep networks. Benefited from end-to-end training and multiple recursive inputs, our networks can achieve better performance than VD2D3D while only using 2D EM images.

3 Multi-stage Multi-recursive-input Fully Convolutional Networks

3.1 Overview

Fig. 2 illustrates the proposed network architecture. Our networks consist of multiple sequential stages, where each stage is a sub-net built based on fully convolutional networks with multiple side outputs [42]. Each sub-net consists of multiple levels, each of which is composed of several combinations of one convolutional layers followed by one ReLU layer and a final pooling layer. Each side output layer is connected to the last convolutional layer of each level, which is composed of a $1\times 1$ convolutional layer and a deconvolutional layer to ensure the resolution of each side output is the same as the input original image. A sigmoid layer is applied to each side output layer to generate a neuronal boundary map having values belonging to $[0,1]$.

Note that, the multiple side outputs of one stage are concatenated with the original image along the image channel dimension and fed into the next stage. By feeding multiple side outputs to the next stage, the side outputs of next stage can obtain the information from all scales, so that even low level side outputs in the next stage can capture large objects. While in a single stage network, each side output only corresponds to a certain scale, e.g., low level side outputs cannot suppress large intracellular structures, and high level side outputs cannot locate thin membranes accurately. This is like how a human visual system may compare different possible segmentation solutions and select the right scale from them [13]. The multiple stages in our networks can be trained in an end-to-end fashion, which enable the side outputs in a previous stage to receive feedback from those of the next stage, like V4 in the human visual system gives the top-down influence to V1.

3.1.1 Sub-net Architecture

We adopt the well-known HED network [42] as our default sub-net, which is converted from VGG-16 net [37], having 5 levels, with strides 1, 2, 4, 8 and 16, respectively, and receptive field sizes 5, 14, 40, 92, 196, respectively. Each side output layer is connected to the last convolutional layer of each level, i.e., conv1_2, conv2_2, conv3_3, conv4_3, conv5_3, respectively.

3.2 $\textbf{M}^{2}$FCN for Neuronal Boundary Detection

Now we formulate our approach for neuronal boundary detection as a per-pixel classification problem. Given a raw input EM image $X=(x_{j},j=1,\ldots,|X|)$, where index $j$ is over the image spatial dimensions of image $X$, the goal is to predict the neuronal boundary map $\hat{Y}=(\hat{y}_{j},j=1,\ldots,|X|)$, where $\hat{y}_{j}\in\{0,1\}$ denotes the predicted label for each pixel $x_{j}$, i.e., if $x_{j}$ is predicted as a boundary pixel, $\hat{y}_{j}=0$; otherwise, $\hat{y}_{j}=1$. We learn an $\text{M}^{2}$FCN, which consists of $M$ stages and each stage has $N$ levels, to address this problem. Next, we introduce the training and testing phases of our approach respectively.

3.2.1 Training Phase

Since our networks use holistic image training, we consider each training image independently. Suppose that we are given a training batch with one 2D EM image $X$ and its corresponding ground truth neuronal boundary map $Y$, our goal is to supervise the multiple side outputs at different level to approach the ground truth map $Y$. Let $S^{m,n}=(s^{m,n}_{j},j=1,\ldots,|X|)$ be the side output at the $n$-th level of the $m$-th stage. Since the side outputs of one stage will be fed to the next stage as the inputs, we express the input to the $m$-stage by

[TABLE]

where $\oplus$ is the the concatenation operation along the image channel dimension and we define $X^{(1)}=X$. Let $\mathbf{W}^{m}$ be the network parameters for the sub-net in the $m$-th stage and $\mathbf{w}^{m,n}$ be the parameters for the $n$-th side output in the $m$-th stage, we define a cross-entropy loss function for this side output by

[TABLE]

This loss function (Eqn.3.2.1) is computed over all pixels in the training image $X$, where $|B|$ and $|\bar{B}|$ denote the boundary and non-boundary ground truth label sets respectively, $\sigma(\cdot)$ is the sigmoid function and $\beta=|\bar{B}|/|B|$ is a positive/negative class-balancing weight [42] to eliminate the bias between boundary and non-boundary ground truths in training (in a typical 2D EM image, most of the ground truth is non-boundary). To supervise all the side outputs in our network, we define a loss function by

[TABLE]

where $\mathbf{W}=(\mathbf{W}^{m},m=1,\ldots,M)$, $\mathbf{w}=(\mathbf{w}^{m,n},m=1,\ldots,M,n=1\ldots,N)$ and $\alpha_{m,n}$ is a loss weight for each side output.

To obtain a fused output $S^{f,m}$ for $m$-stage , we use a $1\times 1$ convolutional layer to fuse its side outputs:

[TABLE]

where $\mathbf{h}=(h_{m,n},m=1,\ldots,M,n=1\ldots,N)$ is the fusion weight. Similar to Eqn. 3.2.1, a class-balanced cross-entropy loss function is defined for the fused output of $m$-stage:

[TABLE]

where $\mathbf{w}^{m}=(\mathbf{w}^{m,n},n=1\ldots,N)$. The loss function for all the fused outputs is

[TABLE]

where $\alpha_{f,m}$ is a loss weight for each stage.

All these parameters, $\mathbf{W},\mathbf{w},\mathbf{h}$, are optimized simultaneously by standard back-propagation:

[TABLE]

3.2.2 Testing Phase

In the testing stage, given an image $X$, by considering the $m$-th sub-net as a function $F_{m}$, we sequentially obtain the side outputs of the $m$-th sub-net by

[TABLE]

The fusion output of $m$-stage is obtained by

[TABLE]

We use the fused output of the last stage $S^{f,M}$ as the final output of our network, then the unified boundary probability map is given by $\hat{Y}=\sigma(S^{f,M})$.

3.2.3 Initialization for Multi-stage Training

The proposed network is very deep (our 3-stage network with 5 levels has totally 48 layers). It is known that, training such a deep network from scratch is not easy. Here, we adopt a simple strategy to initialize the multi-stage network. First, we train a single stage network. Then, this network is used to initialize the first stage of our network, while the rest of the network is randomly initialized. The initialization for the first stage provides high-confidence recursive inputs to the consecutive stage, which facilitates the training procedure.

4 Experimental Results

In this section, we discuss our designs for network architectures and training strategies, and compare our performance with other competitors.

4.1 Experiment Setting

The hyper parameters of our networks include: the mini-batch size (1), the loss weight for each side-output (1), the momentum (0.9), and the weight decay ($2\times 10^{-4}$). We set the base learning rate to 1e-8 and pre-train a single stage network by 20,000 iterations. Then we use this pre-trained single stage network to initialize the first stage of our multi-stage network, and reduce the base learning rate to 1e-9 and train it by 10,000 iterations.

4.2 Mouse Piriform Cortex Dataset

The images of mouse piriform cortex dataset [21] were collected from the piriform cortex of an adult mouse, which contains 4 stacks of EM images. We use the same training-testing split in [21], i.e., stack2, stack3 and stack4 are for training and stack1 is for testing.

4.2.1 Evaluation Metric

To evaluate membrane segmentation performances, we follow the protocol used in [21], where the segmented membrane is measured by the Rand F-score:

[TABLE]

where $V_{merge}^{Rand}$ and $V_{split}^{Rand}$ are Rand merge score and Rand split score respectively, and defined by:

[TABLE]

where $n_{ij}$ denote the number of voxels in the $i$-th segment of the proposal segmentation and $j$-th segment of the ground truth segmentation. $V_{merge}^{Rand}$ and $V_{split}^{Rand}$ are close to $1$ when there are fewer merge and split errors, respectively. To calculate the Rand score, we obtain the neuronal segmentation based on the boundary map by applying the same modified graph-based watershed algorithm [43] as in [21]. We use the default setting in [21] and report our best Rand F-score $V_{Fscore}^{Rand}$.

4.2.2 Data Augmentation

Data augmentation is a standard way to generate sufficient training data for learning a “good” deep network. We rotate the images to 4 different angles ($0^{\circ}$, $90^{\circ}$, $180^{\circ}$, $270^{\circ}$) and flip them with different axis (up-down, left-right, no flip), then resize images to 3 different scales ($0.8$, $1.0$, $1.2$), totally leading to an augmentation factor of 36.

4.2.3 Alternative Design Discussion

We use the pre-trained single stage network as the baseline and discuss some possible alternative designs for network architectures and training strategies. The results of these alternative designs are summarized in Table. 1. To simplify description, we denote each alternative design by “AD” plus an index.

The role of multiple stages.

Since our networks consist of multiple stages, it’s necessary to see whether the performance can be improved by adding stage by stage. Due to the limitation of GPU memory, the deepest network we can train is a 3-stage network with sub-nets of 5 levels (Sec. 3.1.1). We compare the performance between a 1-stage network (AD_I), a 2-stage network (AD_VI) and a 3-stage network (AD_VII). Note that, the AD_I has the same architecture as the pre-trained single stage network. As shown in Table. 1, the performance of AD_I is almost the same as that of the pre-trained single stage network, which indicates that training a single stage network by more iterations cannot improve the performance considerably. The AD_VI and the AD_VII achieve $1.39\%$ and $1.86\%$ performance improvements compared with the baseline, respectively, which shows that our multi-stage training is effective for neuronal boundary detection. A qualitative comparison between these three networks, i.e, 1-stage (AD_I), 2-stage (AD_VI) and 3-stage (AD_VII) networks, is given in Fig. 3. Observed that, the false detections on intracellular structures such as mitochondria and vesicles can be reduced (indicated by red arrows) by training a network with more stages.

The role of multiple recursive inputs.

As we stated in the Sec. 1, using multiple recursive inputs for the multiple stage training is crucial for our framework. To evaluate this, we test two alternative network architectures, which only uses the side output of the 5-th level (AD_III) and the 4-th level (AD_IV) in each stage as the recursive input for the next stage respectively. As shown in Table. 1, only using single recursive input results in a significant performance drop, $4.56\%$ decrease for AD_III and $2.1\%$ decrease for AD_IV. We illustrate the side outputs of the second stages of AD_III and AD_VI in Fig. 4, where we see that the side outputs of the former one only response to large scale objects, while even the side outputs of the latter one can capture objects of different scales, thanks to the multiple recursive inputs.

Stepwise or end-to-end training.

One difference between our multi-stage training framework to others [39, 21, 14, 9] is we train these multiple stages in an end-to-end fashion, not stepwise. To verify which way is better, we also train a 2-stage network stepwise, by simply fixing the parameters of the first stage in this 2-stage network (AD_V). As can be seen from Table. 1, training the 2-stage network stepwise leads to a considerable performance decrease ($0.9819\rightarrow 0.9762$). During end-to-end training, previous stages are influenced by next stages. We visualize the fused outputs (after watershed) of the first stages of (AD_V) and (AD_VI), respectively, in Fig. 5, where we see the latter one leads to better segmentation results (indicated by red arrows). Therefore, we conclude that training in an end-to-end way is better.

The range of the receptive field sizes in each sub-net.

The receptive field sizes of the 5 levels in each sub-net range from 5 to 196. Such a wide range of receptive field sizes ensure the side outputs to be able to capture small neuronal boundaries while suppress relative big intracellular structures. To show this wide range of receptive field sizes is important for neuronal boundary detection, we use an alternative network architecture for each sub-net, which is obtained by removing the last level from the default sub-net, i.e., a 2-stage network with 4-level sub-net (AD_II). As we expected, as shown in Table. 1, removing the last level in each sub-net leads to performance decrease.

4.2.4 Performance Comparison

Now we compare our networks (3-stage with 5 levels) with other competitors on the mouse piriform cortex dataset. The quantitative results are summarized in Table 2 and the precision (rand merge)-recall (rand split) curves are illustrated in Fig. 6. As can be seen, our method can maintain a high precision even when it achieves a high recall, thanks to the multi-stage training which suppresses false detections of boundaries on intracellular structures while enhancing neuronal boundaries. The current state-of-the-art method on the mouse piriform cortex dataset is VD2D3D [21], which is also a recursive training framework. But it trains two stages stepwise, i.e., train the first one, a 2D convolutional network, then uses its output as the recursive input for the second one, a 3D convolutional networks. The experimental results show that with the end-to-end multi-stage training and multi-recursive-inputs, our 2D 2-stage network can achieve better performance than a 2D-3D network. Note that, as VD2D3D already obtained a high Rand F-score, our method achieves around $1.5\%$ improvement on it, which is meaningful. Such a low error obtained on a large EM image dataset is important for neuron reconstruction. Some neuron segmentation results obtained by applying the watershed algorithm [43] to our boundary maps are shown in Fig. 7.

4.3 ISBI 2012 EM segmentation dataset

Most of current neuronal boundary detection methods are evaluated on the public dataset of ISBI 2012 EM segmentation challenge [29]. The training data of this dataset is a set of 30 consecutive images (512 $\times$ 512 pixels) from a serial section Transmission Electron Microscopy (ssTEM) dataset of the Drosophila first instar larva ventral nerve cord [4]. The testing data of this dataset also contains 30 consecutive EM images of the same resolution. The ground truth boundary maps of the training images are made publicly available to enable participants to develop their algorithm, while the ground truth boundary maps of the test images are kept by the organizers. Although this challenge is over, it is still open for submissions. The performance of the new submissions will be reported on the leader board of this challenge. There are over 70 results listed on the leader board, but not all of them have published papers. We summarized some leading quantitative results reported in published papers in Table 3. Note that, many state-of-the-art methods apply post-processing or average multiple trained models to boost the performance, such as PolyMtl [10], FusionNet [28] and CUMedVision [5]. Our method, a two-stage network using a single trained model without post-processing, can achieve 0.9780 Rand F-score, which is comparable with the state-of-the-art methods and better than CUMedVision [5], a one-stage HED. But, CUMedVision used post-processing and averaged 6 trained models to improve the result. This comparison shows the effectiveness of our multi-stage learning. IAL IC [2] is a post-processing method, which can be applied to our result to improve our performance. Besides, as both FusionNet and PolyMtl are built on ResNet [12], we can also replace the sub-net in our model by such a powerful network to gain improvement.

5 Conclusion

We present multi-stage multi-recursive-input fully convolutional networks for neuronal boundary detection. In the proposed architecture, the multiple side outputs learned at different scales in one stage, are fed into the next stage. This provides the ability to detect neuronal boundaries while suppressing false predictions on intracellular structures. Extensive analysis on two public EM segmentation datasets, the mouse piriform cortex dataset and the ISBI 2012 EM dataset, verifies the advantages of our network architecture.

Acknowledgement. This work was supported in part by the National Natural Science Foundation of China No. 61672336 and No. 61303095, in part by “Chen Guang” project supported by Shanghai Municipal Education Commission and Shanghai Education Development Foundation No. 15CG43, and in part by NSF CCF-1231216.