Leveraging Domain Knowledge to Improve Microscopy Image Segmentation with Lifted Multicuts

TL;DR

This paper introduces a novel graph partitioning approach that incorporates domain-specific biological knowledge to significantly enhance the accuracy of microscopy image segmentation, addressing limitations of current neural network methods.

Contribution

It presents a method to integrate domain knowledge into the lifted multicut framework, improving segmentation accuracy in electron microscopy images.

Findings

Significant accuracy improvements in EM segmentation tasks.

Effective use of long-range interactions informed by biological knowledge.

Enhanced segmentation results over traditional neural network approaches.

Abstract

The throughput of electron microscopes has increased significantly in recent years, enabling detailed analysis of cell morphology and ultrastructure. Analysis of neural circuits at single-synapse resolution remains the flagship target of this technique, but applications to cell and developmental biology are also starting to emerge at scale. The amount of data acquired in such studies makes manual instance segmentation, a fundamental step in many analysis pipelines, impossible. While automatic segmentation approaches have improved significantly thanks to the adoption of convolutional neural networks, their accuracy still lags behind human annotations and requires additional manual proof-reading. A major hindrance to further improvements is the limited field of view of the segmentation networks preventing them from exploiting the expected cell morphology or other prior biological knowledge which humans use to inform their segmentation decisions. In this contribution, we show how such domain-specific information can be leveraged by expressing it as long-range interactions in a graph partitioning problem known as the lifted multicut problem. Using this formulation, we demonstrate significant improvement in segmentation accuracy for three challenging EM segmentation problems from neuroscience and cell biology.

Tables

Table 1. Table 1 : Variants of our approach compared to the method of (Krasowski et al., 2017 ) . The Rand Error measures the over-all segmentation quality, while VI-Split measures the degree of over-segmentation and VI-Merge the degree of under-segmentation. For all measures, a lower score corresponds to a better segmentation.

Method	VI-Split	VI-Merge	Rand Error
MC (Krasowski et al., 2017)	0.3471	0.6347	0.0787
AMWC (Krasowski et al., 2017)	0.4578	0.4935	0.0754
LMC-D	0.4144	0.4445	0.0891
LMC-S	0.4133	0.3788	0.0362
LMC-S (No Mitos)	0.4038	0.3966	0.0363
LMC-SI	0.5054	0.3998	0.0586

Table 2. Table 2 : Results on the drosophila medulla dataset. We compare the segmentation results of Multicut (MC), Lifted Multicut solved for the whole volume (LMC-SI) and Lifted Multicut solved separately for all sub-problems arising from falsely merged objects (LMC-SI) with the results of FFN form (Januszewski et al., 2018 ) . We use a cutout for validation and evaluate with the complete ground-truth segmentation (Full) and a subset of closely proof-read objects (Whitelist).

	Full		Whitelist
	VI-split	VI-merge	VI-split	VI-merge
MC	1.5246	1.9057	1.2189	0.6532
LMC-S	1.6110	0.9405	1.3050	0.2544
LMC-SI	1.5773	0.5403	1.2369	0.0122
FFN	1.4653	0.6340	0.8702	0.0559

Table 3. Table 3 : Quality of the sponge chonanocyte segmentation for the three different types of structures.

Method	VI-Split	VI-merge	Rand Error
Cells
MC	0.6058	0.0116	0.0783
LMC	0.6004	0.0116	0.0782
Flagella
MC	0.4728	0.0812	0.1205
LMC	0.2855	0.0812	0.0429
Microvilli
MC	3.1760	1.1101	0.7409
LMC	2.2745	1.1807	0.6973

Table 4. Table 4 : Comparison of Multciut and Lifted Multicut segmentation results for two time points taken from the light-sheet root primoridia data.

	MC			LMC-S
	VI-split	VI-merge	Rand Error	VI-Split	VI-merge	Rand Error
Timepoint 45	0.3596	0.5918	0.1641	0.3740	0.5527	0.1517
Timepoint 49	0.4586	0.7116	0.2019	0.5153	0.5485	0.1873

Table 5. Table 5 : Overview of the three problem set-ups. RF abbreviates random forest.

	Normal Edges	Dense Lifted Edges	Sparse Lifted Edges
Drosophila Neural Tissue	Mean boundary evidence	-	\pbox20 cmFalse merge
oracle predictions
Murine Neural Tissue	RF based on edge features	\pbox20 cmRF based on region/
clustering features	\pbox20 cmRF based on axon/
dendrite attribution
Sponge Choanocytes	Mean boundary evidence	-	\pbox20 cmsemantic segmentation of
small structures
Arabidopsis Roots	Mean boundary evidence	-	\pbox20 cminstance segmentation
of nuclei

Table 6. Table 6 : Evaluating our proposed hierarchical solver and other multicut solvers. In order to run this experiment, we have constructed a smaller lifted multicut problem from the Drosophila neural tissue dataset by cutting out a 1 × \times 10 × \times 10 micron block from its center, computing graph and local edge weights as described in section 4 , introducing dense lifted edges within a graph neighborhood of 2 and setting their costs to the most repulsive edge cost along the weighted shorted path between the edge’s terminal nodes. The problem at hand contained approximately 34,000 nodes, 244,000 normal edges and 2,384,000 lifted edges. The evaluation shows that the proposed solver yields energies comparable to Kernighan-Lin or Fusion-Moves, but its runtime is two orders of magnitude smaller and comparable to Greedy-Additive (which yields inferior energies). Kernighan-Lin was warm-started with the results of Greedy-Additive and Fusion-Moves with the results of Kernighan-Lin. Hierarchical has used Kernighan-Lin (warm-started with the solution of Greedy-Additive) for the sub-problems. While we only compare the solvers for a single problem size, we have observed very good scalability of our solver, which has solved much larger problems in section 4 .

	Energy	Time [s]
Greedy-Additive (Beier and others., 2016)	-1585593.5	2.03
Kernighan-Lin (Keuper et al., 2015)	-1645876.7	174.69
Fusion-Moves (Beier and others., 2016)	-1645876.7	181.48
Hierarchical (Ours)	-1630274.3	3.29

Equations

$$w_e=\text{log}\frac{1-p_e}{p_e}\forall e\in E.$$

$$\underset{y_e\in Y_E}{\min }\sum _{e\in E}{w}_e{y}_e\text{under the constraints}$$

$$\forall C\in \text{cycles}(G)\forall e\in C:y_e\le \sum _{\hat{e}\in C\setminus \{e\}}{y}_{\hat{e}},$$

$$\underset{y_e\in Y_{EF}}{\min }\sum _{e\in E\cup F}{w}_e{y}_e\text{under the constraints}$$

$$\forall C\in \text{cycles}(G)\forall e\in C:y_e\le \sum _{\hat{e}\in C\setminus \{e\}}{y}_{\hat{e}}$$

$$\forall vw\in F\forall P\in vw-\text{paths}(G):y_{vw}\le \sum _{e\in P}{y}_e$$

$$\forall vw\in F\forall c\in vw-\text{cuts}(G):1-y_{vw}\le \sum _{e\in C}(1-y_e).$$

Full text

The amount of data acquired in such studies makes manual instance segmentation, a fundamental step in many analysis pipelines, impossible. While automatic segmentation approaches have improved significantly thanks to the adoption of convolutional neural networks, their accuracy still lags behind human annotations and requires additional manual proof-reading. A major hindrance to further improvements is the limited field of view of the segmentation networks preventing them from learning to exploit the expected cell morphology or other prior biological knowledge which humans use to inform their segmentation decisions. In this contribution, we show how such domain-specific information can be leveraged by expressing it as long-range interactions in a graph partitioning problem known as the lifted multicut problem. Using this formulation, we demonstrate significant improvement in segmentation accuracy for four challenging boundary-based segmentation problems from neuroscience and developmental biology.

1 Introduction

Large-scale electron microscopy (EM) imaging is becoming an increasingly important tool in different fields of biology. The technique was pioneered by the efforts to trace the neural circuitry of small animals at synaptic resolution to obtain their so-called connectome – a map of neurons and synapses between them. In the 1980’s White et al. (White et al., 1986) have mapped the complete connectome of C. elegans in a manual tracing effort which spanned over a decade. Since then, the throughput of EM imaging has increased by several orders of magnitude thanks to innovations like multi-beam serial section EM (Eberle et al., 2015), hot-knife stitching (Hayworth et al., 2015) or gas cluster milling (Hayworth et al., 2018). This allows to image much larger volumes up to the complete brain of the fruit-fly larva (Eichler et al., 2017) and even the adult fruit-fly (Zheng et al., 2018). Recently, studies based on large-scale EM have become more common in other fields of biology as well (Nixon-Abell et al., 2016; Otsuka et al., 2018; Russell et al., 2017).

In light microscopy, very large image volumes became routine even earlier (Royer et al., 2016; Krzic et al., 2012; Keller et al., 2008), with Terabyte-scale acquisitions not uncommon for a single experiment. While the question of segmenting cell nuclei at such scale with high accuracy has been addressed before (Amat et al., 2014), cell segmentation based on membrane staining remains a challenge and a bottleneck in analysis pipelines.

Given the enormous amount of data generated, automated analysis of the acquired images is crucial; one of the key steps being instance segmentation of cells or cellular organelles. In recent years, the accuracy of automated segmentation algorithms has increased significantly thanks to the rise of deep learning-based tools in computer vision and the development of convolutional neural networks (CNNs) for semantic and instance segmentation (Beier et al., 2017; Funke et al., 2018b; Lee et al., 2017; Januszewski et al., 2018). Still, it is not yet good enough to completely forego human proof-reading. Out of all microscopy image analysis problems, neuron segmentation in volume EM turned out to be particularly difficult (Januszewski et al., 2018) due to the small diameter and long reach of neurons and astrocytes, but other EM segmentation problems have not yet been fully automated either. Heavy metal staining used in the EM sample preparation affects all cells indiscriminately and forces segmentation algorithms to rely on membrane detection to separate the objects. The same problem arises in the analysis of light microscopy volumes with membrane staining, where methods originally developed for EM segmentation also achieve state-of-the-art results (Funke et al., 2018a).

One of the major downsides of CNN-based segmentation approaches lies in their limited field of view which makes them overly reliant on local boundary evidence. Staining artifacts, alignment issues or noise can severely weaken this evidence and often cause false merge errors where separate objects get merged into one. On the other hand, membranes of cellular organelles or objects of small diameter often cause false split errors where a single structure gets split into several segmented objects.

Human experts avoid many of these errors by exploiting additional prior knowledge on the expected object shape or constraints from higher-level biology. Following this observation, several algorithms have recently been introduced to enable detection of morphological errors in segmented objects (Rolnick et al., 2017; Zung et al., 2017; Dmitriev et al., 2018; Matejek et al., 2019). By looking at complete objects rather than a handful of pixels, these algorithms can significantly improve the accuracy of the initial segmentation. In addition to purely morphological criteria, Krasowski et al. in (Krasowski et al., 2017) suggested an algorithm to exploit biological priors such as an incompatible mix of ultrastructure elements.

Building on such prior work, this contribution introduces a general approach to leverage domain-specific knowledge for the improvement of segmentation accuracy. It allows to incorporate a large variety of rules, explicit or learned from data, which can be expressed as the likelihood of certain areas in the image to belong to the same object in the segmentation. The areas can be sparse and/or spatially distant. In more detail, we formulate the segmentation problem with such rules as a graph partitioning problem with long-range attractive or repulsive edges.

For the problem of image segmentation, the graph in the partitioning problem corresponds to the region adjacency graph of the image pixels or superpixels. The nodes of the graph can be mapped directly to spatial locations in the image. When domain knowledge can be expressed as rules that certain locations must or must not belong to the same object, it can be distilled into lifted (long-range) edges between these locations. The weights of such lifted edges are derived from the strictness of the rules, which can colloquially range from ”usually do / do not belong to the same object” to ”always / never ever belong to the same object”.

Horvnakova et al. in (Horňáková et al., 2017) showed that this problem, which they term Lifted Multicut as it corresponds to the Multicut partitioning problem with additional edges between non-adjacent nodes, can be solved exactly in reasonable time for small problem sizes, while Beier et al. in (Beier and others., 2016) introduced an efficient approximate solver.

In the following, we demonstrate the versatility of the Lifted Multicut based-approach by applying it to four segmentation problems, three in EM and one in LM. We incorporate starkly different kinds of prior information into this framework:

• •

Based on the knowledge that axons are separated from dendrites in mammalian cortex, we use indicators of axon/dendrite attribution to avoid merges between axonal and dendritic neural processes (Figure 1(left));

• •

Based on the knowledge of plausible neuron morphology, we correct false merge errors in the segmentation of neural processes (Figure 1(center));

• •

Based on the knowledge that certain biological structures form long continuous objects, we reduce the number of false splits in instance segmentation of sponge choanocytes (Figure 1(right));

• •

Based on the knowledge that a cell should only contain one nucleus, we improve the segmentation of growing plant lateral roots (Figure 4).

Aiming to apply the method to data of biologically relevant size, we additionally introduce a new scalable solver for the lifted multicut problem based on our prior work from (Pape et al., 2017). Our code is available at https://github.com/constantinpape/cluster_tools.

2 Related Work

Neuron segmentation for connectomics has been the main driver of the recent advances in boundary-based segmentation for microscopy. Most methods ((Andres et al., 2012; Beier et al., 2017; Nunez-Iglesias et al., 2013; Funke et al., 2018b; Lee et al., 2017)) follow a three step procedure: in the first step they segment boundaries, in the second compute an over-segmentation into superpixels and finally agglomerate the superpixels into objects.

The success of a CNN (Ciresan et al., 2012) in the ISBI 2012 neuron segmentation challenge (Arganda-Carreras et al., 2015) has prompted the adoption of this technique for the boundary prediction step. Most recent approaches use a U-Net (Ronneberger et al., 2015) architecture and custom loss functions (Lee et al., 2017; Funke et al., 2018b). The remaining differences between methods can be found in the superpixel merging procedure. Several approaches are based on hierarchical clustering, but differ in how they accumulate boundary weights: (Lee et al., 2017) use the accumulated mean CNN boundary predictions, (Funke et al., 2018b) employ quantile based accumulation and (Nunez-Iglesias et al., 2013) re-predict the weights with a random forest (Breiman, 2001) after each agglomeration step. In contrast (Andres et al., 2012) and (Beier et al., 2017) solve a NP-hard graph partitioning problem, the (Lifted) Multicut. Notable exception from this three step approach are the flood filling network (FFN) (Januszewski et al., 2018) and MaskExtend (Meirovitch et al., 2016) which can go directly from pixels to instances by predicting individual object masks one at a time, as well as 3C (Meirovitch et al., 2018), which can simultaneously predict multiple objects.

Krasowski et al (Krasowski et al., 2017) showed that the common three-step procedure can be modified to incorporate sparse biological priors at the superpixel agglomeration step. They use the Asymmetric Multi-Way Cut (AMWC) (Kroeger et al., 2014), a generalization of the Multicut for joint graph partition and node labeling. The method is based on exploiting the knowledge that, given the field of view of modern electron microscopes, axon- and dendrite-specific ultrastructure should not belong to the same segmented objects in mammalian cortex. While this approach can be generalized to other domain knowledge, it has two important drawbacks. First, it is not possible to encode attractive information just with node labels. Second, it is harder to express information that does not fit the node labeling category, even if it is repulsive in nature. A good example for this is the morphology-based false merge correction. In this case, defining a labeling for only a subset of nodes is not possible.

Lifted Multicut formulation has been used for neuron segmentation by (Beier et al., 2017). However, the lifted edges were added densely and their weights and positions were not based on domain knowledge, but learned from groundtruth segmentations by the Random Forest algorithm. Only edges over a graph distance of 3 were considered. These lifted edges made the segmentation algorithm more robust against single missing boundaries, but did not counter the problem of the limited field of view of the boundary CNN and did not prevent biologically implausible objects. Note that this approach can be seen as a special case of the framework proposed here, using generic, but weak knowledge about local morphology and graph structure of segments. Besides Lifted Multicut, the recently introduced Mutex Watershed (Wolf et al., 2018, 2019) and generalized agglomerative clustering (Bailoni et al., 2019) can also exploit long-range information.

While all the listed methods demonstrate increased segmentation accuracy, they do not offer a general recipe on how to exploit domain-specific knowledge in a segmentation algorithm. We propose a versatile framework that can incorporate such information from diverse sources by mapping it to sparse lifted edges in the lifted multicut problem.

3 Methods

Our method follows the three step segmentation approach described in section 2, starting from a boundary predictor and using graph partitioning to agglomerate super-pixels. First, we review the lifted multicut problem (Horňáková et al., 2017) in subsection 3.1. We follow by proposing a general approach to incorporate domain-specific knowledge into the lifted edges (subsection 3.2). Finally, we describe four specific applications with different sources of domain knowledge and show how our previous work on lifted multicut for neuron segmentation can be positioned in terms of the proposed framework.

3.1 Lifted Multicut Graph Partition

Instance segmentation can be formulated as a graph partition problem given a graph $G=\{V,E\}$ and edge weights $W\in]-\infty,\infty[$ . In our setting, the nodes $V$ correspond to fragments of the over-segmentation and edges $E$ link two nodes iff the two corresponding fragments share an image boundary. The weights $W$ encode the attractive strength (positive values) or repulsive strength (negative values) of edges and are usually derived from (pseudo) probabilities $P$ via negative log-likelihood:

[TABLE]

The resulting partition problem is known as multicut or correlation clustering (Chopra et al., 1993; Andres et al., 2012; Demaine et al., 2006). Its objective is given by

[TABLE]

where $Y_{E}$ are binary indicator variables linked to the edge state; 0 means that an edge connects the two adjacent nodes in the resulting partition, 1 means that it doesn’t. The constraints forbid dangling edges in the partition, i.e. edges that separate two nodes ($y_{e}=1$) for which a path of connecting edges ($y_{e}=0$) exists.

The lifted multicut (Horňáková et al., 2017) is an extension of the multicut problem, which introduces a new set of edges $F$ called lifted edges. These edges differ from regular graph edges by providing only an energy contribution, but not inducing connectivity. This is motivated by the observation that it is often helpful to derive non-local features for the connectivity of (super) pixels. The presence of an attractive non-local edge should not result in air bridges though, i.e. non-local edges that connect two pixels without a connection via local edges. In our setting, lifted edges connect nodes $v$ and $w$ that are not adjacent in $G$. With the sets of original edges $E$, lifted edges $F$, binary indicator variables $Y$, and weights $W$ associated with all edges in ${E\cup F}$ the lifted multicut objective can be formulated as

[TABLE]

The constraints (5) correspond to Equation 2 and enforce a consistent partition without dangling edges. Constraints (6) and (7) ensure that the state of lifted edges is consistent with the connectivity, i.e. that two nodes connected by a lifted edge are also connected via a path of regular edges and two nodes separated by a lifted edge are not connected by any such path.

3.2 Sparse Lifted Edges

Our main contribution is a general recipe how to express domain-specific knowledge via sparse lifted edges that are only added between graph nodes where attribution of this knowledge is possible. The right part of Figure 2 shows a sketch of this idea: nodes with attribution are shown by shaded segments and sparse lifted edges by yellow dashed lines.

The sparse lifted edges are constructed in several steps, see Figure 1. We compute the superpixels by running the watershed algorithm on boundary predictions and construct the corresponding region adjacency graph. Figure 1(b) shows regular, not lifted, edges between superpixels, green for attractive and red for repulsive weights. Then, we map the domain specific knowledge to nodes as shown in Figure 1(c), and derive attractive and repulsive lifted edges, again shown as green and red lines in (d). The sign and strength of the lifted edge weights can be learned or introduced explicitly, reflecting the likelihood of incident nodes being connected. Equation 1 is used to obtain signed weights. Finally, we solve the resulting lifted multicut objective to obtain an instance segmentation, shown in Figure 1(e).

Mouse Cortex Segmentation, EM

This application example shows how the framework described above can be used to incorporate the axon/dendrite attribution priors first introduced in (Krasowski et al., 2017). We detect the axon- and dendrite-specific elements and map them to the nodes in the same way as (Krasowski et al., 2017) (Figure 1(c), with blue shading for axon and yellow for dendrite attribution). The difference comes in the next step: instead of introducing semantic node labels for ”axon” and ”dendrite” classes, we add repulsive lifted edges between nodes which got mapped differently. subsection 4.1 includes more details on the problem set-up and results.

Drosophila brain segmentation, EM

For neurons in the insect brain, the axon/dendrite separation is not pronounced and the approach described in the previous section can not be applied directly. Instead, morphological information can be used to identify and resolve errors in segmented objects. This was first demonstrated by (Rolnick et al., 2017), where a CNN was trained on downsampled segmentation masks to detect merge errors. Meirovitch et al. in (Meirovitch et al., 2016) detect merge errors with a simple shape-based heuristic and then correct these with a MaskExtend algorithm. Zung et al. (Zung et al., 2017) were the first to combine CNN-based error detection and flood filling network-based correction. In their formulation both false merges and false splits can be corrected. Recently (Dmitriev et al., 2018; Matejek et al., 2019) have introduced an approach based on CNN error detection followed by a simple heuristic to correct false merges and lifted multicut graph partitioning to fix false splits.

Based on all this prior work which convincingly demonstrates that false merge errors can be detected in a post-processing step, we concentrate our efforts on error correction, emulating the detection step with an oracle. We extract skeletons for all segmented objects and have the oracle predict, for all paths connecting terminal nodes of a skeleton, if this path goes through a false merge location (passes through an unidentified boundary). Note that the oracle is not perfect and we evaluate the performance of the algorithm for different levels of oracle error.

If the oracle predicts the path to go through a false merge, we introduce a repulsive lifted edge between the terminals of the path. The weights of the edges are also predicted by the oracle. Figure 1 shows an example of this approach: the red object in the middle of panel (c) has been detected as a false merge. The corresponding lifted edges are shown in Figure 1(d).

Sponge segmentation, EM

In this example, we tackle a segmentation problem in a sponge choanocyte chamber. These structures are built from several surrounding cells, the choanocytes, that interact with a central cell via flagella which are surrounded by a collar of microvilli. Our goal is to segment cell bodies, flagella and microvilli. This task is challenging due to the large difference in sizes of these structures. Especially the segmentation of the small flagella and microvilli is difficult. Without the use of domain specific knowledge on their continuity, the Multicut algorithm splits them up into many pieces.

In order to alleviate these false split errors, we predict which pixels in the image belong to flagella and microvilli and compute an approximate flagella and microvilli instance segmentation via thresholding and connected components. We map the component labels to nodes of the graph, see right column in Figure 1(c). Then, we introduce attractive lifted edges between the nodes that were covered by the same component and repulsive lifted edges between nodes mapped to different components, see Figure 1(d).

Lateral root segmentation, LM

Finally, we tackle a challenging segmentation problem in light-sheet data: segmentation of root cells in Arabidopsis thaliana. This data was imaged with two channels, showing cell membrane and nucleus markers. We use the first channel to predict cell boundaries and the second to segment individual nuclei. The nuclei then serve as bases to force each segmented cell to only contain one nucleus: we introduce repulsive lifted edges between nodes which are covered by different nuclei instances. subsection 4.4 shows how this setup helps prevent false merge errors in cell segmentation.

3.3 Hierarchical Lifted Multicut Solver

Finding the optimal solution of the lifted multicut objective is NP-hard. Approximate solvers based on greedy algorithms (Keuper et al., 2015) and fusion moves (Beier et al., 2017) have been introduced. However, even these approximations do not scale to the large problem we need to solve in the sponge segmentation example. In order to tackle this and even larger problems, we adapt the hierarchical multicut solver of (Pape et al., 2017) for lifted multicuts.

This solver extracts sub-problems from a regular tiling of the volume, solves these sub-problems in parallel and uses the solutions to contract nodes in the graph, thus reducing its size. This approach can be repeated for an increasing size of the blocks that are used to tile the volume, until the reduced problem becomes feasible with an other (approximate) solver.

We extend this approach to the lifted multicut by also extracting lifted edges during the sub-problem extraction. We only extract lifted edges that connect nodes in the sub-graph defined by the block at hand. This strategy, where we ignore lifted edges crossing block boundaries, is in line with the idea that lifted edges contribute to the energy, but not to the connectivity. Note that lifted edges that are not part of any sub-problem at a given level will still be considered at a later stage. See appendix algorithm 1 for pseudo-code. The comparison to other solvers in appendix Table 6 shows that it indeed scales better to large data. Note that this approach is conceptually similar to the fusion move based approximation of (Beier and others., 2016), which extracts and solves sub-problems based on a random graph partition and accepts changes from sub-solutions if they increase the overall energy, repeating this process until convergence. Compared to this approach, we extract sub-problems from a deterministic partition of the graph. This allows us to solve only a preset number of sub-problems leading to faster convergence.

Note that our approximate solver is only applicable if the graph at hand has a spatial embedding, which allows to extract sub-problems from a tiling of space. In our case, this spatial embedding is given by the watershed fragments that correspond to nodes.

4 Results

We study the performance of the proposed method on four different problems: i) neuron segmentation in murine cortex with priors from axon/dendrite segmentation, ii) neuron segmentation in drosophila brain with priors from morphology-based error detection, iii) instance segmentation in a sponge choanocyte chamber with priors from semantic classes of segmented objects, ix) cell segmentation in Arabidopsis roots with priors from ”one nucleus per cell” rule. Appendix Table 5 summarizes the different problem set-ups. We evaluate segmentation quality using the variation of information (VI) (Meilă, 2003), which can be separated into split and merge scores, and the adapted rand score (Arganda-Carreras et al., 2015). For all error measures used here, a lower value corresponds to higher segmentation quality.

4.1 Mouse Cortex Segmentation, EM

We present results on a volume of murine somatosensory cortex that was acquired by FIBSEM at 5 $\times$ 5 $\times$ 6 nanometer resolution. The same volume has already been used in (Krasowski et al., 2017) for a similar experiment. To ensure a fair comparison between the two methods for incorporating axon/dendrite priors, we obtained derived data from the authors and use it to set-up the segmentation problem.

This derived data includes probability maps for cell membrane, mitochondria, axon and dendrite attribution as well as a watershed over-segmentation derived from the cell membrane probabilities and ground-truth instance segmentation. From this data, we set up the graph partition problem as follows: we build the region adjacency graph $G$ from the watersheds and compute weights for the regular edges with a random forest based on edge and region appearance features. See (Beier et al., 2017) for a detailed description of the feature set. Next, we introduce dense lifted edges up to a graph distance of three. We use a random forest based on features derived from region appearance and clustering to predict their weights, see (Beier et al., 2017) for details. In addition to the region appearance features only based on raw data, we also take into account the mitochondria attribution here. Next, we map the axon/dendrite attribution to the nodes of $G$ and introduce sparse lifted edges between nodes mapped to different classes. We infer weights for these edges with a random forest based on features from the statistics of the axon and dendrite node mapping. We use the fusion move solver of (Beier and others., 2016) for optimizing the lifted multicut objective.

We divide the volume into a 1 $\times$ 3.5 $\times$ 3.5 micron block that is used to train the random forests for edge weights and a 2.5 $\times$ 3.5 $\times$ 3.5 micron block used for evaluation. The random forest predicting pixel-wise probabilities was trained by the authors of (Krasowski et al., 2017) on a separate volume, using ilastik (Sommer et al., 2011).

We compare the multicut and AMWC solutions reported in (Krasowski et al., 2017) with different variants of our methods, see Table 1. As a baseline, we compute the lifted multicut only with dense lifted edges and without features from mitochondria predictions (LMC-D). We compute the full model with dense and sparse lifteds (LMC-S) with and without additional features for dense lifted edges from mitochondria predictions. In addition, we compare to an iterative approach (LMC-SI) similar to the error correction approach in subsection 4.2, where we perform LMC-D segmentation first and introduce sparse lifted edges only for objects that contain a false merge (identified by presence of both axonic and dendritic nodes in the same object).

The LMC-D segmentation quality is on par with the AMWC, although it does not use any input from the priors, showing the importance of dense lifted edges. Our full model with sparse lifted edges shows significantly better quality compared to LMC-D. Mitochondria-based features provide a small additional boost. The segmentation quality of the iterative approach LMC-SI is inferior to solving the full model LMC-S. This shows the importance of joint optimization of the full model with dense and sparse lifted edges.

4.2 Drosophila brain segmentation, EM

We test the false merge correction on parts of the Drosophila medulla using a 68 $\times$ 38 $\times$ 44 micron FIBSEM volume imaged at 8 $\times$ 8 $\times$ 8 nanometer from (Takemura et al., 2015), who also provide a ground-truth segmentation for the whole volume.

First, we train a 3D U-Net for boundary prediction on a separate 2 $\times$ 2 $\times$ 2 micron cube. We use this network to predict boundaries on the whole volume, and run watershed over-segmentation based on these predictions. Then, we set up an initial Multicut with edge weights derived from mean accumulated boundary evidence. We obtain an initial segmentation by solving it with the block-wise solver of (Pape et al., 2017).

In order to demonstrate segmentation improvement based on morphological features, we skeletonize all sufficiently large objects using the method of (Lee et al., 1994) implemented in (Van der Walt et al., 2014). We then predict false merges along all paths between skeleton terminal nodes, using the groundt-ruth segmentation as oracle predictor. Note that (Dmitriev et al., 2018) have shown that it is possible to train a very accurate CNN to classify false merges based on morphology information in this set-up. Given these predictions, we set up the Lifted Multicut problem by selecting all objects that have at least one path with a false merge detection. For these objects, we introduce lifted edges between all terminal nodes corresponding to paths and derive weights for these edges from the false merge probability (note that we use an imperfect oracle for some experiments, so the merge predictions are not absolutely certain). We solve two different variants of this problem, LMC-S, where we solve the whole problem using the solver introduced in subsection 3.3 and LMC-SI, where we only solve the sub-problems arising for the individual objects. For this, we use the Fusion Moves solver of (Beier and others., 2016).

Table 2 compares the results of the initial Multicut (MC) with LMC-S and LMC-SI (using a perfect oracle) as well as the current state of the art FFN based segmentation (Januszewski et al., 2018). We adopt the evaluation procedure of (Januszewski et al., 2018) and use a cutout of size 23 $\times$ 19 $\times$ 23 micron for validation. We use two different versions of the ground-truth, the full segmentation and only a set of white-listed objects that were more carefully proofread. The FFN segmentation and validation ground-truth was kindly provided by the authors of (Januszewski et al., 2018). The results show that our initial segmentation is inferior to FFN in terms of merge errors, but using LMC-SI we can improve the merge error to be even better than FFN. Interestingly, LMC-SI performs better than LMC-S. We suspect that this is due to the fact that we only add lifted edges inside of objects with a false merge detection, thus LMC-S does not see more information then LMC-SI, while having to solve a much bigger optimization problem.

In Figure 3 we show the initial segmentation and three corrected merges. Panel (e) evaluates LMC-S and LMC-SI on the full ground-truth when using an imperfect oracle: we tune the oracle’s F-score from 0.5 to 1.0 and measure VI-split and VI-merge. The curves show that LMC-SI is fairly robust against noise in the oracle predictions; it starts with a lower VI-merge than the initial MC, even for F-Score 0.5 and its VI-split gets close to the MC value for F-Score 0.75+.

4.3 Sponge segmentation, EM

The two previous experiments mostly profited from repulsive information derived from ultrastructure or morphology. In order to show how attractive information can be exploited, we turn to an instance segmentation problem in a sponge choanocyte chamber. The EM volume was imaged with FIBSEM at a resolution of 15 $\times$ 15 $\times$ 15 nanometer. We aim to segment structures of three different types: cell bodies, flagella and microvilli. Flagella and microvilli have a small diameter, which make them difficult to segment with a boundary based approach. On the other hand, cell bodies have a much larger diameter and touch each other, which makes a boundary based approach appropriate.

In order to set-up the segmentation problem, we first compute probability maps for boundaries, microvilli and flagella attribution using the autocontext workflow of ilastik (Sommer et al., 2011). We set-up the lifted multicut problem by first computing watersheds based on boundary maps, extracting the region adjacency graph and computing regular edge weights from the boundary maps accumulated over the edge pixels. We do not introduce dense lifted edges. For sparse lifted edges, we compute an additional instance segmentation of flagella and microvilli by thresholding the corresponding probability maps and running connected components. Then, we map the components of this segmentation to graph nodes and connect nodes mapped to the same component via attractive lifted edges and nodes mapped to different components via repulsive lifted edges. We use the hierarchical lifted multicut solver introduced in subsection 3.3 to solve the resulting objective, using the approximate solver of (Keuper et al., 2015) to solve sub-problems. Note that the full model contained too many variables to be optimized by any other solver in a reasonable amount of time.

We run our segmentation approach on the whole volume, which covers a volume of 70 $\times$ 75 $\times$ 50 microns, corresponding to 4600 $\times$ 5000 $\times$ 3300 voxels. For evaluation, we use three cutouts of size 15 $\times$ 15 $\times$ 1.5 microns with ground-truth for instance and semantic segmentation. We split the evaluation into separate scores for objects belonging to the three different structures, extracting them based on the semantic segmentation ground-truth. See Table 3 for the evaluation results, comparing the sparse lifted multicut (LMC) to the multicut baseline (MC). As expected the quality of the segmentation of cell bodies is not affected, because we don’t introduce lifted edges for those. The split rate in flagella and microvilli decreases significantly leading to a better overall segmentation for these structures.

4.4 Lateral root segmentation, LM

We segment cells in light-sheet image volumes of the lateral root primordia of Arabidopsis thaliana. The time-lapse video consisting of 51 time points was obtained in vivo in close-to-natural growth conditions. Each time point is a 3D volume of size 2048 $\times$ 1050 $\times$ 486 voxels each with resolution 0.1625 $\times$ 0.1625 $\times$ 0.25 micron. The volume has two channels, one showing membrane marker, the other nucleus marker. We work on two selected time points, namely: $T_{45}$ and $T_{49}$ taken from the later stages of development where the instance segmentation problem is more challenging due to growing number of cells. The time points have dense ground-truth segmentation for a 1000 $\times$ 450 $\times$ 200 voxels cutout centered on the root primordia. Both cells and nuclei ground truth are available.

A variant of 3D U-Net (Çiçek et al., 2016) was trained in order to predict cell membranes and nuclei respectively. The two networks were trained on dense ground-truth from time points which were not part of our evaluation. Apart from the primary task of predicting membranes and nuclei respectively, both networks were trained on an auxiliary task of predicting long-range affinities similarly to (Lee et al., 2017) which proved to improve the effectiveness of the main task.

Using these networks, we predict cell boundary probabilities and nucleus foreground probabilities. We use the nucleus predictions to obtain a nucleus instance segmentation by thresholding the probability maps at $p_{threshold}=0.9$ and running connected components analysis.

We compute superpixel from the watershed transform on the membrane predictions and compute weights for the regular edges via mean accumulated boundary evidence. We set up lifted edges by mapping the nucleus instances to superpixels and connecting all nodes whose superpixels were mapped to different nuclei with repulsive lifted edges.

Table 4 shows the evaluation of segmentation results on the ground-truth cutouts. We can see that LMC-S clearly improves the merge errors as well the overall Rand Error while only marginally diminishing the split quality. See Figure 4 for an overview of the qualitative results.

5 Discussion

We propose a general purpose strategy to leverage domain-specific knowledge for instance segmentation problems arising from EM image analysis. This strategy makes use of a graph partitioning problem known as lifted multicut by expressing the domain knowledge in the long-range lifted edges. We apply the proposed strategy to a diverse set of instance segmentation problems in light and electron microscopy and consistently show an improvement in segmentation accuracy. For an application with ultrastructure based priors, we also observe that the lifted multicut based formulation yields higher quality results than the AMWC formulation of (Krasowski et al., 2017). We believe that this is due to joint exploitation of dense short-range and sparse long-range information. A complete joint solution, with both lifted edges and semantic labels, has recently been introduced in (Levinkov et al., 2017). We look forward to exploring the potential of this objective for the neuron segmentation problem.

Similar to the findings of (Kroeger et al., 2014), we demonstrated that prevention of merge errors is more efficient than their correction: the joint solution of LMC-S is more accurate than iterative LMC-SI. However, not all prior information can be incorporated directly into the original segmentation problem. For these priors we demonstrate how to construct an additional resolving step which can also significantly reduce the number of false merge errors. In the future we plan to further improve our segmentations by other sources of information: matches of the segmented objects to known cell types, manual skeletons or correlative light microscopy imaging.

6 Acknowledgements

We gratefully acknowledge the support of the Baden-Wuerttemberg Stiftung, and the contributions to this work made by Klaske J. Schippers and Nicole L. Schieber in the Electron Microscopy Facility of EMBL.

7 Appendix

7.1 Overview of problem set-up

7.2 Hierarchical Lifted Multciut Solver