Efficient optimization for Hierarchically-structured Interacting Segments (HINTS)

TL;DR

This paper introduces a novel optimization algorithm called Path-Moves for hierarchical segmentation models with geometric interactions, enabling effective use of complex tree-structured label hierarchies in biomedical image segmentation.

Contribution

The paper presents the Path-Moves algorithm, a new optimization method capable of handling arbitrary tree-structured hierarchies with interactions, overcoming limitations of existing methods.

Findings

Achieves state-of-the-art biomedical segmentation results.

Handles complex tree hierarchies with geometric segment interactions.

Outperforms previous optimization approaches in accuracy and applicability.

Abstract

We propose an effective optimization algorithm for a general hierarchical segmentation model with geometric interactions between segments. Any given tree can specify a partial order over object labels defining a hierarchy. It is well-established that segment interactions, such as inclusion/exclusion and margin constraints, make the model significantly more discriminant. However, existing optimization methods do not allow full use of such models. Generic -expansion results in weak local minima, while common binary multi-layered formulations lead to non-submodularity, complex high-order potentials, or polar domain unwrapping and shape biases. In practice, applying these methods to arbitrary trees does not work except for simple cases. Our main contribution is an optimization method for the Hierarchically-structured Interacting Segments (HINTS) model with arbitrary trees. Our Path-Moves algorithm is based on multi-label MRF formulation and can be seen as a combination of well-known a-expansion and Ishikawa techniques. We show state-of-the-art biomedical segmentation for many diverse examples of complex trees.

Tables

Table 1. Table 1: Compares our Path-Moves optimization to QPBO [ 25 ] and a-exp [ 7 ] which were proposed by [ 8 ] . The precision and recall were averaged over 15 examples. Our method and QPBO clearly outperformed a-exp which was very sensitive to initialization and the order in which labels were expanded on. On average QPBO left 2.8 % percent 2.8 2.8\% of the pixels unlabeled and in one instance 7 % percent 7 7\% . These values rise significantly when not using the Hedgehog shape prior, see Table 2 .

	Grey-Matter			White-Matter			CSF			SGM
	Ours	QPBO	a-exp	Ours	QPBO	a-exp	Ours	QPBO	a-exp	Ours	QPBO	a-exp
F1 Score	$0.92$	$0.83$	$0.32$	$0.92$	$0.90$	$0.56$	$0.85$	$0.82$	$0.04$	$0.83$	$0.81$	$0.37$
Precision	$0.87$	$0.87$	$0.88$	$0.92$	$0.92$	$0.46$	$0.78$	$0.83$	$0.02$	$0.92$	$0.93$	$0.23$
Recall	$0.97$	$0.80$	$0.19$	$0.93$	$0.88$	$0.74$	$0.93$	$0.82$	$0.56$	$0.76$	$0.71$	$0.92$

Table 2. Table 2: Compares optimization methods when using min-margins constraints without Hedgehog shape prior. X 𝑋 X implies that the corresponding label was not assigned to any pixel in the final solution. The precision and recall were averaged over 15 examples. Our method clearly outperformed QPBO and a-exp. On average QPBO left 8.5 % percent 8.5 8.5\% of the pixels unlabeled and in one instance 11.5 % percent 11.5 11.5\% .

	Grey-Matter			White-Matter			CSF			SGM
	Ours	QPBO	a-exp	Ours	QPBO	a-exp	Ours	QPBO	a-exp	Ours	QPBO	a-exp
F1 Score	$0.92$	$0.53$	$X$	$0.93$	$0.90$	$0.58$	$0.84$	$0.77$	$0.05$	$0.84$	$0.79$	$0.02$
Precision	$0.87$	$0.82$	$X$	$0.93$	$0.95$	$0.47$	$0.77$	$0.84$	$0.03$	$0.90$	$0.90$	$0.11$
Recall	$0.97$	$0.39$	$X$	$0.93$	$0.85$	$0.76$	$0.92$	$0.71$	$0.50$	$0.79$	$0.71$	$0.91$

Table 3. Table 3: Weighted prec. and recall were averaged over 7 test cases. All methods preformed comparably when using Hedgehogs [ 13 ] .

	Ours	QPBO	a-exp
F1 Score	$0.95$	$0.95$	$0.93$
Weighted Precision	$0.95$	$0.95$	$0.94$
Weighted Recall	$0.95$	$0.95$	$0.92$

Table 4. Table 4: Weighted precision and recall were averaged over 7 test cases. Eliminating Hedgehog priors [ 13 ] lead to a significant decrease in segmentation quality. On average QPBO left 10.8 % percent 10.8 10.8\% of the pixels unlabeled.

	Ours	QPBO	a-exp
F1 Score	$0.88$	$0.85$	$0.82$
Weighted Precision	$0.92$	$0.93$	$0.89$
Weighted Recall	$0.85$	$0.78$	$0.76$

Equations

$$E(\mathbf{f})=\stackrel{\text{emdata}}{\overbrace{\sum _{p\in \Omega }{D}_p(f_p)}}+\stackrel{\text{emsmoothness}}{\overbrace{\lambda \sum _{pq\in \mathcal{N}}{V}_{pq}(f_p,f_q)}}+\stackrel{\begin{array}{c}\text{eminteraction}\\ \text{emconstraints}\end{array}}{\overbrace{T(\mathbf{f})}}$$

$$V_{pq}(\alpha ,\beta )=\sum _{ij\in \Gamma (\alpha ,\beta )}{V}_{pq}(i,j),$$

$$T(\mathbf{f})=w_{\infty }\sum _{\begin{array}{c}\ell \in \mathcal{L}\end{array}}\sum _{\begin{array}{c}p\in \Omega \\ f_p\in \mathcal{T}(\ell )\end{array}}\sum _{\begin{array}{c}q\in \Omega \\ \Vert p-q\Vert <{\delta }_{\ell }\end{array}}[f_q\notin \{\mathcal{T}(\ell )\cup \mathcal{P}(\ell )\}]$$

$$S(\mathbf{f})=w_{\infty }\sum _{\ell \in \mathcal{L}}\sum _{\begin{array}{c}p\in \Omega \\ f_p\in \mathcal{T}(\ell )\end{array}}\sum _{pq\in {\mathcal{S}}_{\ell }}[f_q\notin \mathcal{T}(\ell )],$$

$$\!\!\!\!D_{p}(f_{p})=\!\left\{\begin{array}[]{ll}\!\!\!-\ln(Pr(I_{p}|f_{p}))&\text{background}\\ \!\!\!-\ln(Pr(I_{p}|f_{p}))+DT(p)&\text{otherwise,}\end{array}\right.$$

$$\text{weighted precision}=\sum _{\ell \in \mathcal{L}}\frac{|{\mathbf{f}}^{\ast }=\ell |}{|\Omega |}{\text{precision}}_{\ell }\text{vspace}-1ex$$

$$\begin{gathered} cost=V(a_i,a_{i-1})+V(a_{i-1},a_{i-2})+\dots \\ +V(a_{i+x+1},a_{i+x}). \end{gathered}$$

$$\begin{gathered} cost=V(a_i,a_{i-1})+\dots +V(a_2,a_1) \\ +V(a_1,c_n)+\dots +V(c_{t-1},c_t). \end{gathered}$$

$$\begin{array}{rl}cost=& \stackrel{V(a_i,a_1)}{\overbrace{V(a_i,a_{i-1})+V(a_2,a_1)}}+\dots \\ & +\underset{V(a_1,c_t)}{\underbrace{V(a_1,c_n)+\dots +V(c_{t-1},c_t)}}\\ =& V(a_i,c_t),\end{array}\text{vspace}-2ex$$

$$\begin{array}{rl}cost=& \stackrel{V(b_j,a_1)}{\overbrace{V(b_j,b_{j-1})+\dots +V(b_m,a_1)}}\\ & +\underset{V(a_1,a_i)}{\underbrace{V(a_1,a_2)+\dots +V(a_{i-1},a_i)}}\\ =& V(b_j,a_i),\end{array}$$

$$\begin{array}{rl}cost=& \stackrel{V(b_j,a_1)}{\overbrace{V(b_j,b_{j-1})+\dots +V(b_m,a_1)}}\\ & +\underset{V(a_1,c_t)}{\underbrace{V(a_1,c_n)+\dots +V(c_{t-1},c_t)}}\\ =& V(b_j,c_t),\end{array}$$

Full text

1 Introduction

Basic cues like smooth boundaries and appearance models are often insufficient to regularize complex segmentation problems. This is particularly true in medical applications where objects have weak contrast boundaries and overlapping appearances. Thus, additional priors are needed, e.g. shape-priors [28], volumetric constraints [3], or segments interaction [30, 8]. The latter constraint is the essence of Hierarchically-structured Interacting Segments (HINTS) model111HINTS model was proposed by [8] by was not given a name. [8, 30], which was successfully applied to many segmentation problems, e.g. cells [22], joint cartilage [30], cortical [23] or tubular [20] surfaces.

HINTS model overview:​​​

Any hierarchically-structured222We use hierarchically-structured and partially ordered interchangeably. segments could be represented as a label tree $\mathcal{T}$, see Fig. 1. Tree $\mathcal{T}$ defines topological relationship between segments as follows; (a) child-parent relation means the child segment is inside its parent’s segment, (b) sibling relation means the corresponding segments exclude each other. For example, in Fig. 1 $A$ is inside $R$, $E$ is inside $B$, while $B$, $C$ and $D$ exclude each other in $A$. Min-margin is one form of interaction between regions. If region $X$ has $\delta_{X}$ min-margin then its outside boundary pushes away the outside boundary of its parent and siblings by at least $\delta_{X}$, Fig.1(b).

Limitations of previous algorithms:​​​

We extend [8], which introduced HINTS for arbitrary trees. In [8] a-expansion (a-exp) [7] was used to optimize the multi-label formulation of HINTS, but it often results in bad local minima due to complexities of interaction constraints, e.g. Fig.2. The contribution of [8] is a binary multi-layered HINTS formulation. They use high-order data terms, which are not easy to convert into unary and pairwise potentials for arbitrary trees. Their algorithm’s global optimality guarantee depends on the tree at hand. Only trees that do not yield frustrated cycles [25] have this guarantee, but this is not immediately obvious for any given tree. In [8], non-submodular binary energy implied by frustrated cycles were addressed by QPBO [25]. In practice, QPBO produces only partial solutions for most trees, see Figs. 2, 15 and 17.

As an alternative to QPBO, [27] formulated HINTS as constraint optimization. They solve the Lagrangian dual of this NP-hard problem using an iterative sequence of graph cuts. However, the duality gap may be arbitrarily large and the optimum for HINTS is not guaranteed. Their super-gradient optimization of Lagrange multiplier guesses initial solutions and time-step parameters. Also, according to Lemma 5 in [27] their super-gradient corresponds to the hard exclusion constraint of HINTS, which is $\{0,\infty\}$-valued. They do not discuss how this affects the algorithm.

In [30] the authors generalize their earlier method [20] for segmenting multiple nested surfaces, i.e. $\mathcal{T}$ is a chain. In [30] the aim was to segment multiple mutually exclusive objects each with a set of nested surfaces, i.e. $\mathcal{T}$ is a spider333Tree with one node of degree $\geq 3$ and all others with degree $\leq 2$.. But, the proposed approach can handle only a single pair of mutually exclusive objects in a given image region. As such [30] requires a prior knowledge of the region of interaction for two excluded objects, or computes it using a problem specific trained classifier [30]. Such prior knowledge is not required for our method. In contrast to our approach, [30] requires a sufficiently close initial segmentation satisfying interaction constraints. In all of our experiments we started from a trivial solution. Unlike our approach, [30] implicitly imposes a star like shape prior [28] and use non-homogeneous anisotropic polar grids.

If interactivity (min-margin) constraints are dropped HINTS degenerates to tree-metric labeling. Certain tree-metric labeling problems are addressed in [10] using DP to find the global optima if the data terms are also a tree-metric. Recently, [1] used convex relaxation to approximate labeling problems where labels are leafs of a DAG. This problem can be reduced444Personal communication with the authors of [1]. to general metric labeling [17]. Such labeling problems are significantly different from HINTS due to interactions between segments.

Motivation for Path-Moves:​​​

In the context of multi-label HINTS formulation, we propose an effective move-making algorithm applicable to arbitrary label trees $\mathcal{T}$ avoiding limitations of the previous optimization methods.

In contrast to a-exp [7], our Path-Moves are non-binary: when expanding label $\alpha$ any pixel can change its current label to any label along the path connecting its current label and $\alpha$ in the tree. Optimization uses our generalization of the well-known multi-layered Ishikawa technique [15] for convex potentials over strictly ordered labels. In essence, Path-Moves combine a-exp and Ishikawa. In the special case of a chain-tree our algorithm reduces to Ishikawa-like construction in [8, 15] finding global minimum in one step. On the contrary, when $\mathcal{T}$ is a single-level star our algorithm reduces to a-exp. Note that closely related multi-label range-moves [29] also combine a-exp and Ishikawa for non-convex pairwise potentials over strictly ordered labels (a chain). In contrast to Path-Moves, in range-moves all pixels have the same set of feasible labels.

Our contributions are summarized below:

• •

we propose Path-Moves - approximate optimization method applicable to HINTS. Unlike [8, 30], Path-Moves work for arbitrary trees avoiding weak local minima typical of a-exp [7] in the context of HINTS.

• •

we show how a generalization of star shape priors, e.g. [28, 12, 13], integrate into multi-label HINTS model, if needed. Path-Moves can address this too.

• •

we show state-of-the-art biomedical segmentation results for complex trees.

2 Hierarchically-structured Interacting Segments

Given pixel set $\Omega$, neighborhood system $\mathcal{N}$, and labels (regions) $\mathcal{L}$ the HINTS model can be formulated as

[TABLE]

where $f_{p}$ is a label assigned to $p$ and $\mathbf{f}=\left[f_{p}\in\mathcal{L}|\;\forall p\in\Omega\right]$ is a labeling of all pixels.

The data and smoothness terms are widely used in segmentation, e.g. [6, 4]. Data term $D_{p}(f_{p}$) is the cost incurred when pixel $p$ is assigned to label $f_{p}$. Usually, $D_{p}$ is negative log likelihood of the label’s probabilistic model, which could be fitted using scribbles [4, 24] or known a priori.

The smoothness term regularizes segmentation discontinuities. A discontinuity occurs when two neighboring pixels $(p,q)\in\mathcal{N}$ are assigned to different labels. Parameter $\lambda$ weights the importance of the smoothness term. The most commonly used smoothness potential is Potts model [7].We use tree-metric smoothness [10, 11] which is more true to the physical structure of the labels in some settings, especially medical segmentation as we explain shortly.

A function $V$ is tree-metric if there exists a tree with non-negative edge weights and $V(u,w)$ is equal to the sum of edge weights along the unique path between nodes $u$ and $w$ in the tree. In our setting the label tree $\mathcal{T}$ is such a tree and $V_{pq}$ is completely defined by assigning non-negative weights to every edge in $\mathcal{T}$. Thus, for any $\alpha$, $\beta$ in $\mathcal{L}$

[TABLE]

where $\Gamma({\alpha},{\beta})$ is the set of ordered labels on the path between $\alpha$ and $\beta$ in the undirected tree $\mathcal{T}$. The summation in (2) is between pairs of neighboring labels on path $\Gamma({\alpha},{\beta}).$

To motivate tree-metric smoothness consider $\mathcal{T}$ in Fig.1(a). This tree implies that regions $R$, $A$ and $D$ are nested. For example, $R$, $A$ and $D$ could be background, cell and nucleus, respectively. In the physical world boundaries of nested regions never merge into a single boundary. That is, if in the image we observe a boundary between $D$ and $R,$ this corresponds to two boundaries, namely $D/A$ and $A/R$ in the physical world. Therefore, the $D/R$ boundary cost should be the sum of $D/A$ and $A/R$ boundary costs. The summation property of nested boundaries can be modeled as tree-metric smoothness. In contrast, Potts model penalizes multiple nested boundaries as a single boundary.

The interaction term in (1) ensures that the min-margin constraints are satisfied at every pixel, see Fig. 3,

[TABLE]

where $w_{\infty}$ is an infinitely large scalar, $\mathcal{T}(X)$ are the nodes of the subtree rooted at $X$, $\mathcal{P}(X)$ is $X$’s parent in $\mathcal{T}$, and $[\;]$ is the Iverson bracket. This term guarantees that any labeling that violates min-margin constraint has infinite energy.

In general, the interaction term could model not only min-margin but also region attraction [8], scene parsing [21, 8], or a combination of these constraints. However, the focus of this paper is developing an effective combinatorial optimization move for energy (1). Thus, for simplicity of exposition we only cover min-margins.

We now compare our formulation to that in [8]. Inclusion is an easy constraint to impose in both formulations as it reduces to using tree-metric smoothness. In our formulation exclusion is satisfied by definition because we use multi-label formulation and each pixel is assigned to only one label. In contrast, in [8] the label of a pixel is represented by several binary variables. Therefore, [8] needs to explicitly enforce exclusion to maintain the validity of these binary variables w.r.t. tree $\mathcal{T}$. Often this leads to non-submodular terms that are difficult to optimize.

3 Optimization

In Section 3.1 we introduce our Path-Move algorithm and in Section 3.2 we show which interaction constraints Path-Move could optimize. The authors in [8] showed that HINTS is non-submodular for a general tree $\mathcal{T}$ and they used either QPBO or a-exp for optimization. Unfortunately, QPBO does not guarantee to label all pixels and we observed that in our experiments, see Fig. 2. The a-exp algorithm [7] is guaranteed to label all pixels but prone to weak local minima, Fig. 2.

We build on a-exp algorithm [7]. The algorithm in [7] maintains a valid current labeling $\mathbf{f}^{\prime}$ and iteratively tries to decrease the energy by switching from the current labeling to a nearby labeling via a binary expansion move. In a binary expansion, a label $\alpha\in\mathcal{L}$ is chosen randomly and allowed to expand. Each pixel is given a binary choice to either stay as $f^{\prime}_{p}$ or switch to $\alpha$, i.e. $f_{p}\in\{f^{\prime}_{p},\alpha\}$. The algorithm stops when it cannot decrease the energy anymore.

.

.

Due to the “binary” nature of the expansion move interaction constraints cause a-exp to be highly sensitive to initialization and prone to converge to a weak local minima even for simple trees, see Fig. 4.

Instead of using a binary expansion move [7] in step 4 of the a-exp algorithm, we propose a more powerful “multi-label” move, namely, Path-Move. Figure 4(d) shows how robust a Path-Move is compared to a binary one [7].

3.1 Path-Move

In a Path-Move on $\alpha$ each pixel $p$ can choose any label in the ordered set $\Gamma({f^{\prime}_{p}},{\alpha})$ where $f^{\prime}_{p}$ is the current label of $p$. Thus, the set of feasible labels for $p$ is $\Gamma({f^{\prime}_{p}},{\alpha})$, see examples in Fig. 5.

Given an arbitrary $\mathcal{T},$ current labeling $\mathbf{f}^{\prime},$ and label $\alpha$, we now show how to build a graph such that the min-cut on this graph corresponds to the optimal Path-Move. We use $s$ and $t$ to denote source and sink nodes of the min-cut problem, respectively. Our construction is motivated by [15, 7, 29].

Data Term:

For each pixel $p$ we generate a chain of nodes ${\cal C}_{p}$ whose size is $|\Gamma({f^{\prime}_{p}},{\alpha})|-1$. Let us rename $\Gamma({f^{\prime}_{p}},{\alpha})$ to $(u_{1},u_{2},\ldots,u_{h})$ where $u_{1}=f^{\prime}_{p}$ and $u_{h}=\alpha$. Note that $u_{i}$ and $h$ depend on $p$ but we drop explicit dependence on $p$ from notation for clarity. Figure 6(a) illustrates chain ${\cal C}_{p}$ and how it is linked to $s$ and $t$. The edge weights along the directed path $(s,{\cal C}_{p},t)$ encode the the data terms of $p$ while the weights along the opposite direction are $w_{\infty}$. If the $i^{th}$ edge along the $(s,{\cal C}_{p},t)$ is cut, then pixel $p$ is assigned to label $u_{i}$. The $w_{\infty}$ edges ensure that any min-cut severs only one edge on the $(s,{\cal C}_{p},t)$ path as proposed by [15]. Thus, the sum of severed edges on paths $(s,{\cal C}_{p},t)$ for all pixels $p\in\Omega$ adds to the data term in (1).

It should be noted that although $D_{p}$ is used as a weight for n-link555An edge between two nodes and neither of those nodes is $s$ or $t$. edges it could still be negative. In case of negative $D_{p}$ a positive constant $K$ is added to $D_{p}(\ell)$ for all $p\in\Omega$ and $\ell\in\mathcal{L}$ to ensure that the new data terms are non-negative for every pixel—equivalent to adding a constant $|\Omega|K$ to (1).

Smoothness:

Let $p$ and $q$ be a pair of neighboring pixels. Note the overlap between $\Gamma({f^{\prime}_{p}},{\alpha})$ and $\Gamma({f^{\prime}_{q}},{\alpha})$ is at least one label, see Fig. 5. Our graph construction treats the sequence of overlapping labels of paths $\Gamma({f^{\prime}_{p}},{\alpha})$ and $\Gamma({f^{\prime}_{q}},{\alpha})$ differently from the non-overlapping parts. Therefore we rename $\Gamma({f^{\prime}_{p}},{\alpha})=(b_{1},\ldots,b_{m},a_{1},\ldots,a_{k})$ and $\Gamma({f^{\prime}_{q}},{\alpha})=(c_{1},\ldots,c_{n},a_{1},\ldots,a_{k})$ to emphasize the overlap. Figure 6(b) shows the newly weighted edges that are added to out constructed graph to encode the smoothness penalty $V_{pq}$.

The overlapping part $(a_{1},\ldots,a_{k})$ forms a linear ordering for which the smoothness cost is encoded as proposed by [15]. The non-overlapping parts $(b_{1},\ldots,b_{m},a_{1})$ and $(c_{1},\ldots,c_{m},a_{1})$ each forms a linear ordering independent of the other, but extending $(a_{1},\ldots,a_{k})$ linear ordering. In this case smoothness penalties are handled by additional edges from $s$, for proof of correctness see Appendix A.

Interaction Constraints:

Let $p$ and $q$ be $\delta_{A}>0$ within each other. As per energy (3), to impose the $\delta_{A}$ margin constraint between $p$ and $q$ we need to add edges to our graph to ensure that whenever $f_{p}\in\mathcal{T}(A)$ and $f_{q}\not\in\{\mathcal{T}(A)\cup\mathcal{P}(A)\}$ the corresponding energy is infinite. Thus, we need to eliminate/forbid labels along the $\Gamma({f^{\prime}_{q}},{\alpha})$ path that would violate the $\delta_{A}$ constraints if $p$ is assigned to a label in $\mathcal{T}(A).$ To impose such constraint between $p$ and $q$ expansion paths there are several cases to consider depending on whether each of $\alpha$, $f^{\prime}_{p}$ or $f^{\prime}_{q}$ is in $\mathcal{T}(A)$ or not as follows.

Scenario I when $\alpha\in\mathcal{T}(A)$:

Case 1, assume $f^{\prime}_{p}\not\in\mathcal{T}(A)$ and $f^{\prime}_{q}\not\in\mathcal{T}(A)$. Since $f^{\prime}_{p}\not\in\mathcal{T}(A)$ and $\alpha\in\mathcal{T}(A)$ by assumption, we can deduce that $A$ and $\mathcal{P}(A)$ are both in $\Gamma({f^{\prime}_{p}},{\alpha})$ see Fig.7 (a), otherwise our assumption that $\mathcal{T}$ is a tree would be violated. Following the same reasoning we can deduce that $A$ and $\mathcal{P}(A)$ are both in $\Gamma({f^{\prime}_{q}},{\alpha})$. Thus, there are possible labelings/configurations involving $p$ and $q$ that violate $\delta_{A}$. In other words, if $p$ is assigned to a label in $\mathcal{T}(A)$ there is a chance of assigning $q$ to a label that is not in $\{\mathcal{T}(A)\cup\mathcal{P}(A)\}$, since part of $q$ expansion path is not entirely in $\{\mathcal{T}(A)\cup\mathcal{P}(A)\}$, Fig. 7(a). We forbid those configurations by adding a $w_{\infty}$ edge between graph chains ${\cal C}_{p}$ and ${\cal C}_{q}$ as shown in Fig.8(a). Thus, eliminating the possibility of a min. cut that would simultaneously assign $q$ to a label $f_{q}\not\in\{\mathcal{T}(A)\cup\mathcal{P}(A)\}$ and $p$ to a label $f_{p}\in\mathcal{T}(A).$

Case 2, assume $f^{\prime}_{q}\in\mathcal{T}(A)$. Since $f^{\prime}_{q}\in\mathcal{T}(A)$ and $\alpha\in\mathcal{T}(A)$ by assumption, we can deduce that $\Gamma({f^{\prime}_{q}},{\alpha})\subseteq\mathcal{T}(A)$, otherwise our assumption that $\mathcal{T}$ is a tree would be violated, see Fig. 7 (b). Thus, additional edges are not needed since $f_{q}$ is guaranteed to be in $\mathcal{T}(A)$. Simply, in this case there is no chance of violating the $\delta_{A}$ constraints regardless of what label is assigned to $p$, because we could only assign $q$ to a label in $\mathcal{T}(A)$, i.e. $f_{q}\in\{\mathcal{T}(A)\cup\mathcal{P}(A)\}$.

Case 3, assume $f^{\prime}_{p}\in\mathcal{T}(A)$ and $f^{\prime}_{q}\not\in\mathcal{T}(A)$. Since $\alpha\in\mathcal{T}(A)$ and $f^{\prime}_{q}\not\in\mathcal{T}(A)$ by assumption, we can deduce that $\mathcal{P}(A)\in\Gamma({f^{\prime}_{q}},{\alpha})$, otherwise $\mathcal{T}$ is not a tree. On the one hand, if $f^{\prime}_{q}=\mathcal{P}(A)$ then no additional edges needed since $f_{q}\in\{\mathcal{P}(A)\cap\mathcal{T}(A)\}$. On the other hand, the case when $f^{\prime}_{q}\not=\mathcal{P}(A)$ is not possible as this would imply that the current labeling violates the $\delta_{A}$. Recall, Path-Moves switches from one valid labeling to another.

Scenario II when $\alpha\not\in\mathcal{T}(A)$:

Case 1, assume $f^{\prime}_{p}\in\mathcal{T}(A)$ and $f^{\prime}_{q}\in\mathcal{T}(A)$. This case follows the same reasoning as scenario I, case 1. Similarly to scenario I, case 1 we handle the forbidden configurations by adding an additional $w_{\infty}$ edge as shown in Fig.8(b).

Case 2, assume $f^{\prime}_{p}\not\in\mathcal{T}(A)$. Since $f^{\prime}_{p}$ and $\alpha$ are $\not\in\mathcal{T}(A)$ by assumption, we can deduce that $f_{p}\not\in\mathcal{T}(A)$. Thus, no new edges are needed, as we are only interested in the case when $f_{p}\in\mathcal{T}(A)$. Recall that we only forbid labels along chain ${\mathcal{C}}_{q}$ if $f_{p}$ is in $\mathcal{T}(A)$, which is not possible in this case.

Case 3, assume $f^{\prime}_{p}\in\mathcal{T}(A)$ and $f^{\prime}_{q}\not\in\mathcal{T}(A)$. Since $f^{\prime}_{q}$ and $\alpha$ are $\not\in\mathcal{T}(A)$ by assumption then we can deduce that $\Gamma({f^{\prime}_{q}},{\alpha})\not\subseteq\mathcal{T}(A).$ On the one hand, if $\mathcal{P}(A)\not\in\Gamma({f^{\prime}_{q}},{\alpha})$ then we can deduce that $f^{\prime}_{q}\not\in\{\mathcal{T}(A)\cup\mathcal{P}(A)\}$ while $f^{\prime}_{p}\in\mathcal{T}(A)$ (by assumption), i.e. current labeling violates the $\delta_{A}$ constraint and this is not possible. On the other hand, if $\mathcal{P}(A)\in\Gamma({f^{\prime}_{q}},{a})$ then we can deduce that $f^{\prime}_{q}=\mathcal{P}(A)$ otherwise the current labeling would violate $\delta_{A}$. When $f^{\prime}_{q}=\mathcal{P}(A)$ the construction is as shown in Fig.8(b) except there are no nodes above $\mathcal{P}(A)$ for $q$.

3.2 Interaction Representability Condition

Unfortunately, not every interaction constraint could be represented as constraints between expansion paths during a Path-Move. To be specific, sometimes interactions lead to conflicting constraints between the expansion paths, i.e. the graph construction inadvertently forbids a permissible configuration. We refer to an interaction that could be optimized by Path-Move as a Path-Move representable constraint, e.g. min-margin is a Path-Move representable constraint while strict box-layout constraints described in [21, 8] for scene parsing is not Path-Move representable.

The objective of box-layout scene parsing is segmenting the scene into 5 regions; left, top, right, bottom, and background, denoted by L, T, R, B, and G, respectively, in Fig. 9(a). The strict box-layout constraints are illustrated in Fig. 9(b-f). For example as shown in (b), when pixel $q$ is directly above $p$ and $f_{p}=G$ then according to the layout in (a) $q$ could only be labeled $G$ or $T$, i.e. $f_{q}\not\in\{L,R,B\}$. The rest of the constraints shown in Fig. 9(b-f) are derived in the same way from the layout in (a).

To show a case that leads to conflicting constraints let us consider the tree shown in Fig. 10(a). Also assume that pixel $p$ is directly below $q$ and that their current labeling is $G$ and $T$, respectively. As shown in Fig. 11(Left), when expanding on $L$ an $w_{\infty}$ edge is add to avoid assigning $p$ and $q$ to $L$ and $G$, respectively, which is a forbidden configuration as per Fig. 9(c). However, the same edge also forbids a permissible labeling that would assign $p$ and $q$ to $L$ and $T$, respectively. As you can see, when using the hierarchical tree in Fig. 10(a) we can not properly represent the strict box-layout interaction constraints during a Path-Move.

In general, an interaction constraint is not Path-Move representable if there exists $\alpha,\beta$ and $\gamma\in\mathcal{L}$ where $a<b\in\Gamma({\gamma},{\alpha})$ and $c<d\in\Gamma({\beta},{\alpha})$ while configuration $[a,d]$ is prohibited and $[b,c]$ is permissible, see Fig. 11 (Right). Nonrepresentable Path-Move interactions lead to a nonsubmodular Path-Move [26]. Nonetheless, this could be avoided either by modifying the tree or relaxing the interaction constraints. For instance, the strict box-layout becomes Path-Move representable when using the alternative hierarchical tree shown in Fig. 10(b). If modifying the hierarchical tree is not an option, then by relaxing the constraints one could always achieve Path-Move representability. For example, by relaxing the strict box-layout constraints shown in Fig. 12 they become Path-Move representable for the hierarchical tree shown in Fig. 10(a).

4 Shape Priors for HINTS

In this section we extend star-shape [28], Geodesic-star [12] and Hedgehogs [13] priors to the HINTS model and show how to enforce these priors during a Path-Move.

In the context of binary segmentation, star-shape prior [28] on label $A$ with star center $c_{A}$ reduces to the following constraint. If pixels $p$ and $q$ lie on any line originating from $c_{A}$ with $q$ in the middle and $p$ is labeled $A$, then $q$ must also be labeled $A$, see Fig.13(a). Geodesic-star [12] and Hedgehogs [13] differ from star-shape prior in terms of what defines the center and how lines from the center (or geodesic paths) are generated. Furthermore, Hedgehogs [13] allow control over shape constraint tightness, see [13] for details.

For partially ordered segments we generalize the star-shape prior constraint as follows. If pixels $p$ and $q$ lie on any line originating from $c_{A}$ with $q$ in the middle and $f_{p}$ is in ${\mathcal{T}}(A)$, then $f_{q}$ must also be in ${\mathcal{T}}(A)$, see Fig.13(b).

The shape prior penalty term is

[TABLE]

where ${\mathcal{S}}_{\ell}$ is the set of all ordered pixel pairs666In practice, it is enough to include only consecutive pixel pairs in ${\cal S}_{\ell}$. $(p,q)$ along any line containing $c_{\ell}$ such that $q$ is between $p$ and $c_{\ell}$. Using [12] or [13] instead of [28] results in a different ${\cal S}_{\ell}$.

Let pixels $p$ and $q$ lie on a line passing through $c_{A}$, and $q$ is between $c_{A}$ and $p$. To impose the star-shape prior for label $A$ during a Path-Move, there are multiple cases to consider depending on whether each of $\alpha$, $f^{\prime}_{p}$ and $f^{\prime}_{q}$ is in $\mathcal{T}(A)$ or not as follows.

Scenario I: when $\alpha\in\mathcal{T}(A)$:

Case 1, assume $f^{\prime}_{p}\not\in\mathcal{T}(A)$ and $f^{\prime}_{q}\not\in\mathcal{T}(A)$. In this case we can deduce that $A$ and $\mathcal{P}(A)$ are both in $\Gamma({f^{\prime}_{p}},{\alpha})$ and $\Gamma({f^{\prime}_{q}},{\alpha})$. Thus, there are possible forbidden configurations and to handle them we add an $w_{\infty}$ edge as in Fig. 14(a).

Cases 2, assume $f^{\prime}_{q}\in\mathcal{T}(A)$. We can deduce that $\Gamma({f^{\prime}_{q}},{\alpha})\subseteq\mathcal{T}(A)$. Thus, no additional edges are needed since $f_{q}$ is guaranteed to be in $\mathcal{T}(A)$.

Case 3, assume $f^{\prime}_{p}\in\mathcal{T}(A)$ and $f^{\prime}_{q}\not\in\mathcal{T}(A)$. An impossible case as the current labeling would be violating the shape-prior.

Scenario II: when $\alpha\not\in\mathcal{T}(A)$:

Case 1, assume $f^{\prime}_{p}\in\mathcal{T}(A)$ and $f^{\prime}_{q}\in\mathcal{T}(A)$. This case is similar to scenario I, case 1, the added edge is shown in Fig. 14(b).

Cases 2, assume $f^{\prime}_{p}\not\in\mathcal{T}(A)$. We can deduce that $\Gamma({f^{\prime}_{p}},{\alpha})\not\subseteq\mathcal{T}(A)$. Thus, no edge needed since $f_{p}$ can not be in $\mathcal{T}(A)$.

Case 3, assume $f^{\prime}_{p}\in\mathcal{T}(A)$ and $f^{\prime}_{q}\not\in\mathcal{T}(A)$, see case 3 above.

5 Experiments

Our 2D medical segmentation experiments focus on comparing Path-Moves for optimizing energy (1) or (1)+(4) to QPBO [25, 8] and a-exp [7, 8]. In all experiments $\lambda$ was set to 1. To define our tree-metric, every edge $(\gamma,\beta)$ in $\mathcal{T}$ was assigned a non-negative weight $V_{pq}(\gamma,\beta)$ computed using a non-increasing function of difference in $p$ and $q$ intensities similar to [2]. Also, whenever a Hedgehog [13] shape prior was used its tightness parameter was set to $\pi/9$.

The experiments evaluate the effectiveness of Path-Moves as a combinatorial multi-labeling move. As such we assume that the color models are known a priori. One can easily integrate Path-Moves in a framework that estimates initial color models using user interaction and iteratively alternates between labeling pixels and re-estimating color models in a GrabCut fashion, e.g. [24, 9, 14].

Brain Segmentation:

We combined the labeled regions in dataset [19] (T1W MRI) to create the tree shown in Fig. 15(a). In this setting, the data term is the sum of color model penalty and an $L_{2}$ shape prior [5] based on an automatically extracted brain mask using [16],

[TABLE]

where $I_{p}$ is the intensity at pixel $p$ and $DT$ is the Euclidean Distance Transform of the extracted brain mask. Min-margins are shown in Fig. 15(a). We also added a Hedgehog prior [13] for the sub-cortical grey-matter to help our energy differentiate between grey-matter and sub-cortical grey-matter.

In this application our method outperformed QPBO in most cases and a-exp in all cases. In fact a-exp always converged to a weak local minima in this setting, see Fig. 15. Based on our experience the quality of a-exp result depends on various factors, e.g. tree complexity, the number of min-margins introduced, the order in which labels are expanded, and the initial solution. For the subjects that QPBO was able to find the global optimal Path-Moves either found the global optimal or a very close solution.

Figure 16 shows the results for Subject 1 with (and without) min-margins and Hedgehog prior. The third row shows the results when not using min-margins. Path-Moves converged after two iterations to a lower energy than a-exp, which converged after six iterations. In this case a-exp local minimum was due to the Hedgehog prior, see last row.

Table 1 compares the precision, recall and F1 score for each region individually, where $F_{1}=2\frac{precision\cdot recall}{precision\;+\;recall}$. The higher F1 values correspond to better segmentation. In general, QPBO was unpredictable as in some cases it found the optimal solution and in other cases it left a large number of pixels unlabeled.

Table 2 show the results after dropping the hedgehog prior. In terms of Path-Moves, the mainly affected label after dropping the Hedgehog prior is the sub-cortical gray matter, as it started to grab parts of the gray matter, see Fig.16 second row, last column. Comparing Tables 1 and 2 it is clear that QPBO and a-exp benefited the most by introducing the hedgehog prior.

Heart Segmentation:

In this setting we only used color models for the data term and no shape priors. Figure 17(a) shows the labels tree. For a-exp to escape its local minimum it needs to first expand the left ventricle and then the left papillary muscles. However, expanding on left ventricle would lead to a higher energy than the current one. Path-Moves avoids this local minimum by allowing both labels to expand simultaneously when performing a Path-Move on the left papillary muscles.

Abdominal Organ Segmentation:

We used CT dataset and extended the work in [13] which used Hedgehogs to segment liver and kidneys. In contrast to [13], we utilized more detailed structures reaching $13$ labels.

For each test case we computed the weighted precision

[TABLE]

where $\mathbf{f}^{*}$ is the ground truth labeling. The weighted recall is defined similarly. As shown in Table 3, all methods performed comparably due to the use of Hedgehog priors and the star-like structure of $\mathcal{T}$, which $a$-exp is well suited for. See Table 4 for results without using Hedgehog priors. Figure 18 shows the tree and our result for one test case. Interestingly, QPBO labeled all the pixels in all 7 test cases. By comparing Tables 3 and 4 it is easy to see the benefit of using Hedgehog priors. Moreover, Path-Moves outperformed QPBO and a-exp after dropping the Hedgehog priors.

We pursued a more challenging structure, see Fig.19(a). The objective in this case was to segment the liver into three segments and any tumors inside them separately. Due to the large overlap between color models and the complex structure, having Hedgehog priors was not enough for QPBO or a-exp to converge to an adequate solution, see Fig.19(c-e). Path-Moves was able to achieve good results by avoiding local minima as in Fig.19(c). Furthermore, Path-Moves guarantees full labeling in contrast to QPBO, which left $7.4\%$ of the pixels unlabeled in Fig.19(d).

In conclusion, our results empirically show that for a general tree a-exp converges to weak local minima, and QPBO is unpredictable in terms of being able to label all pixels. Path-Moves which uses an effective multi-label expansion move, labels all pixels and easily avoids weak local minima that a-exp is prone to.

6 Discussion

Path-Moves is applicable to tree-metrics which could be used to approximate arbitrary metrics [18, 10]. Even in the absence of interactions, Path-Moves is a more powerful move making algorithm than a-exp [7] because of the multi-label nature of its moves. Thus, Path-Moves is a better fit for applications that rely on tree-metrics such as [18]. In the presence of interaction constraints the optimality bound of [7] is not valid. The proof in [7] assumes that given any labeling every pixel with ground truth label $X$ could switch to $X$ via a binary expansion on $X$. This is no longer guaranteed as interaction constraints limit [7] expansion domain, e.g. see Fig.4(c). Our experiments empirically show that Path-Moves finds optimal or near optimal solution. In the cases where QPBO found full labeling, i.e. optimal solution, Path-Moves either found the same solution or a very close one, see Table 3 and Fig.15 Subject 4.

In terms of space complexity a-exp is the most efficient as it requires building a graph with $O(|\Omega|)$ nodes while QPBO requires a significantly larger graph with $O(|\Omega||\mathcal{L}|)$ nodes. A Path-Move graph size depends on $\mathcal{T}$. When $\mathcal{T}$ is balanced it requires $O(|\Omega|\ln(|\mathcal{L}|))$ nodes and $O(|\Omega||\mathcal{L}|)$ in the worse case when $\mathcal{T}$ is a chain.

There is one limitation when using our Path-Moves to optimize (1) compared to [8]. In [8] it is possible to explicitly control the min. exclusion margin between two siblings, say A and B in $\mathcal{T}$. In our model the min. exclusion margin is implicit and it is equal to $\max(\delta_{A},\delta_{B}).$ Because siblings such as $A$ and $B$ are not directly connected in the tree.

Another limitation are interaction constraints that are not Path-Move representable. An interaction constraint is not Path-Move representable if there exists $\alpha,\beta$ and $\gamma\in{\cal L}$ where $a<b\in\Gamma(\gamma,\alpha)$ and $c<d\in\Gamma(\beta,\alpha)$ while configuration $[a,d]$ is prohibited and $[b,c]$ is permissible [26]. In general, this could be avoided either by slightly modifying tree $\cal T$ or relaxing the interaction constraints.

7 Conclusion

The proposed multi-labeling move is effective in optimizing models with hierarchically-structured segments (partially ordered labels) and interaction constraints. In contrast to binary expansion move [7], our move avoids local minima caused by interaction constraints.

Our experiments cover various medical segmentation applications, e.g. brain and heart segmentation. Our results show that Path-Moves always perform at least as well as prior methods. Moreover, Path-Moves significantly outperform prior methods when using complex trees and/or regions with ambiguous color models.

Path-Moves is applicable to arbitrary trees. This is in contrast to [8] which is not easy to generalize for an arbitrary tree as it relies on the cumbersome process of reducing high-order data terms to unary and pairwise potentials.

We generalized star-like shape priors in the context of partially ordered labels. Extending preexisting commonly used priors to partially ordered labels is an interesting idea on its own and we leave this for future work.

Acknowledgments

This work was supported by NIH grants R01-EB004640, P50-CA174521 and R01-CA167632. We thank Drs. S. O’Dorisio and Y. Menda for providing the liver data (NIH grant U01-CA140206). This work was also supported by NSERC Discovery and RTI grants (Canada) for Y. Boykov and O. Veksler.

Appendix A Smoothness Encoding Proof Of Correctness

The following proof uses some of the tree-metric function $V$ properties, mainly symmetry and tree-metric condition (2). Let $p$ and $q$ be a pair of neighbour pixels and assume that we are expanding on label $\alpha\in\mathcal{L}$. Recall that our graph construction treats the sequence of overlapping labels of paths $\Gamma({f^{\prime}_{p}},{\alpha})$ and $\Gamma({f^{\prime}_{q}},{\alpha})$ differently from the non-overlapping parts. Therefore, we rename $\Gamma({f^{\prime}_{p}},{\alpha})=(b_{1},\ldots,b_{m},a_{1},\ldots,a_{k})$ and $\Gamma({f^{\prime}_{q}},{\alpha})=(c_{1},\ldots,c_{n},a_{1},\ldots,a_{k})$ to emphasize the overlap.

To show the correctness of our smoothness encoding shown in Fig.6(b) we need to consider the cost of all possible cuts involving pixels $p$ and $q$. However, it is enough to only consider the four general cuts shown in Fig. 20, which are representative of all possible cuts.

Case I, $f_{p}\in(a_{1},\ldots,a_{k})$ and $f_{q}\in(a_{1},\ldots,a_{k})$: Let $f_{p}=a_{i}$ and $f_{q}=a_{i+x}$ where $1-i\leq x\leq k-i$. If $1-i\leq x<0$ then the cost of severed edges, shown in Fig. 20(a), will be

[TABLE]

Using equation (2) we can deduce that the terms in (6) sum to $V(a_{i},a_{i+x})$, which is the correct smoothness cost based on our assumption regarding $f_{p}$ and $f_{q}$.

If $x=0$ then based on our construction no smoothness edges will be severed between that the $p$ and $q$ expansion paths. Thus, according to our construction the cost for the assigning $p$ and $q$ to $a_{i}$ will be 0, which is correct since $V(\ell,\ell)=0$ for all $\ell\in\mathcal{L}$ by definition.

Finally, the proof when $0<x\leq k-i$ is equivalent to changing the roles of $p$ and $q$ when $1-i\leq x<0$.

Case II, $f_{p}\in(a_{1},\ldots,a_{k})$ and $f_{q}\in(c_{1},\ldots,c_{n})$: Let $f_{p}=a_{i}$ and $f_{q}=c_{t}$. The cost of severed edges, shown in Fig.20(b), will be

[TABLE]

Using equation (2) we can deduce the following

[TABLE]

which is the correct smoothness cost based on our assumption regarding $f_{p}$ and $f_{q}$.

Case III, $f_{p}\in(b_{1},\ldots,b_{m})$ and $f_{q}\in(a_{1},\ldots,a_{k})$: Let $f_{p}=b_{j}$ and $f_{q}=a_{i}$. Using equation (2) we can deduce that the cost of severed edges, shown in Fig.20(c), will be

[TABLE]

which is the correct smoothness cost based on our assumption regarding $f_{p}$ and $f_{q}$.

Case IV, $f_{p}\in(b_{1},\ldots,b_{m})$ and $f_{q}\in(c_{1},\ldots,c_{n})$: Let $f_{p}=b_{j}$ and $f_{q}=c_{t}$. Using equation (2) we can deduce that the cost of severed edges, shown in Fig.20(d), will be

[TABLE]

which is the correct smoothness cost based on our assumption regarding $f_{p}$ and $f_{q}$.

Recall that for any pixel $p$ a non-prohibitively expensive min. cut severs only one edge along its corresponding graph chain $(s,{\mathcal{C}}_{p},t)$, which is due to the data terms exclusion constraints. Thus, this proof showed that our smoothness cost encoding is correct for any feasible cut involving a pair of neighboring pixels.