An Efficient Decomposition Framework for Discriminative Segmentation with Supermodular Losses

TL;DR

This paper introduces an ADMM-based decomposition framework for discriminative image segmentation that enables efficient training with supermodular loss functions, improving accuracy and computational performance.

Contribution

It presents a novel ADMM decomposition method for loss-augmented inference in structured SVMs, allowing practical use of supermodular losses in image segmentation.

Findings

Improved segmentation accuracy on Grabcut and brain data.

Enhanced computational efficiency over existing algorithms.

Validated effectiveness of supermodular losses during training.

Abstract

Several supermodular losses have been shown to improve the perceptual quality of image segmentation in a discriminative framework such as a structured output support vector machine (SVM). These loss functions do not necessarily have the same structure as the one used by the segmentation inference algorithm, and in general, we may have to resort to generic submodular minimization algorithms for loss augmented inference. Although these come with polynomial time guarantees, they are not practical to apply to image scale data. Many supermodular losses come with strong optimization guarantees, but are not readily incorporated in a loss augmented graph cuts procedure. This motivates our strategy of employing the alternating direction method of multipliers (ADMM) decomposition for loss augmented inference. In doing so, we create a new API for the structured SVM that separates the maximum a posteriori (MAP) inference of the model from the loss augmentation during training. In this way, we gain computational efficiency, making new choices of loss functions practical for the first time, while simultaneously making the inference algorithm employed during training closer to the test time procedure. We show improvement both in accuracy and computational performance on the Microsoft Research Grabcut database and a brain structure segmentation task, empirically validating the use of several supermodular loss functions during training, and the improved computational properties of the proposed ADMM approach over the Fujishige-Wolfe minimum norm point algorithm.

Figures

Tables

Table 1. Table 1: The cross comparison of average loss values (with standard error) using the 8-connected loss function ( Δ 8 subscript Δ 8 \Delta_{8} ) and Hamming loss (labeled 0-1) during training. During testing, we evaluate with the Hamming loss, the 8-connected loss and the Intersection over union loss (labeled IoU). Training with the same supermodular loss functions as used during testing yields the best results. Training with supermodular losses even outperforms the Hamming loss in terms of evaluating by Hamming loss.

		Eval.
$\gamma =0.25$		0-1(1e3)	${\Delta }_8$(1e3)	IoU
Train.	0-1	$3.245\pm 0.137$	$6.097\pm 0.267$	$0.2209\pm 0.0075$
	${\Delta }_8$	$3.097\pm 0.141$	$5.807\pm 0.274$	$0.2166\pm 0.0086$

Table 2. Table 2: Wilcoxon sign rank test on the error values in Table  1 comparing training with Hamming loss (labeled 0-1) and the 8-connected loss Δ 8 subscript Δ 8 \Delta_{8} (with different values of γ 𝛾 \gamma ).

	Eval.
	0-1	${\Delta }_8$	IoU
${\Delta }_8,\gamma =0.25$ vs 0-1	$0.0195$	$0.0195$	$0.1055$
${\Delta }_8,\gamma =0.5$ vs 0-1	$0.0371$	$0.0371$	$0.0273$
${\Delta }_8,\gamma =0.75$ vs 0-1	$0.0020$	$0.0020$	$0.0020$
${\Delta }_8,\gamma =1.0$ vs 0-1	$0.0195$	$0.0273$	$0.0488$

Table 3. Table 3: The cross comparison of average loss values on IBSR dataset (cf. comments for Table  1 ).

		Eval.
$\gamma =0.5$		${\Delta }_8$(1e3)	0-1(1e3)	IoU
Train.	${\Delta }_8$	$2.616\pm 0.612$	$1.297\pm 0.224$	$0.169\pm 0.018$
	0-1	$2.885\pm 0.765$	$1.393\pm 0.279$	$0.173\pm 0.019$

Table 4. Table 4: The cross comparison of average loss values (with standard error) using supermodular losses and Hamming loss (labeled 0-1) during training and test time). We additionally evaluate on the intersection over union loss.

		Eval.
		0-1(1e3)	Square loss(1e3) ${\Delta }_{\text{S}}$	Biconvex loss ${\Delta }_{\text{C}}$	IoU loss
Train.	0-1	$3.245\pm 0.137$	$0.257\pm 0.019$	$0.217\pm 0.012$	$0.221\pm 0.007$
	${\Delta }_{\text{S}}$	$2.928\pm 0.418$	$0.251\pm 0.040$	$0.176\pm 0.022$	$0.196\pm 0.016$
	${\Delta }_{\text{C}}$	$2.394\pm 0.166$	$0.202\pm 0.019$	$0.149\pm 0.011$	$0.179\pm 0.007$

Table 5. Table 5: The comparison between ADMM and an LP relaxation for solving the loss augmented inference. The 1st column shows the optimal energy values ( 10 3 superscript 10 3 10^{3} ) (Equation ( 4 )); columns 2–4 show the computation time (s) for one calculation on downsampled images of varying size.

	$-E$	size $=600$	size $=1200$	size $=2400$
ADMM	$2.28\pm 0.58$	$0.035\pm 0.002$	$0.051\pm 0.002$	$0.864\pm 0.476$
LP	$2.29\pm 0.57$	$1.857\pm 0.128$	$3.946\pm 0.286$	$13.57\pm 1.359$

Equations

$$\underset{w,\xi }{\min }\frac{1}{2}\Vert w{\Vert }^2+C\sum _{i=1}^n{\xi }_i$$

$$⟨w,\varphi (x_i,y_i^{\ast })-\varphi (x_i,{\tilde{y}}_i)⟩$$

$$\varphi (x,y):=\left(\begin{array}{c}{\sum }_{j=1}^p{\varphi }_u(x,y^j)\\ {\sum }_{(k,l)\in \mathcal{E}}{\varphi }_p(x,y^k,y^l)\end{array}\right)$$

$$\arg \underset{\tilde{y}\in \mathcal{Y}}{\max }⟨w,\varphi (x,\tilde{y})⟩+\Delta (y^{\ast },\tilde{y}).$$

$$\ell (A\cup \{v\})-\ell (A)\le \ell (B\cup \{v\})-\ell (B).$$

$${\Delta }_8(y^{\ast },\tilde{y})=\sum _{j=1}^p[y^{\ast j}\ne {\tilde{y}}^j]+\sum _{(k,l)\in {\mathcal{E}}_{\ell }}\gamma [y^{\ast k}\ne {\tilde{y}}^k\wedge y^{\ast l}\ne {\tilde{y}}^l]$$

$${\Delta }_{\text{S}}(y^{\ast },\tilde{y})={\left(\frac{{\sum }_{j=1}^p[y^{\ast j}\ne {\tilde{y}}^j]}{\alpha }\right)}^2$$

$$f_a(\cdot ):=f(a,\cdot ):\mathcal{B}\to \mathbb{R}$$

$$f_b(\cdot ):=f(\cdot ,b):\mathcal{A}\to \mathbb{R}$$

$$m:=|y^{\ast }|=\sum _{i=1}^p\left[y_i^{\ast }=+1\right].$$

$$e_{-}:=$$

$$e_{+}:=$$

$$\Delta (y^{\ast },\tilde{y}):=\ell (e_{-},e_{+})$$

$$\hat{\ell }(e_{-},e_{+})=\ell (e_{-},e_{+})\forall e_{-},e_{+}.$$

$${\Delta }_{IoU}(y^{\ast },\tilde{y})=1-\frac{|y^{\ast }\cap \tilde{y}|}{|y^{\ast }\cup \tilde{y}|}.$$

$${\Delta }_C(y^{\ast },\tilde{y})=\frac{|y^{\ast }|+|\tilde{y}|-2|y^{\ast }\cap \tilde{y}|}{|y^{\ast }\cap \tilde{y}|+1}.$$

$${\ell }_C(e_{-},e_{+})=\frac{e_{-}+e_{+}}{m-e_{-}+1}.$$

$${\hat{\ell }}_C(e_{-},e_{+})={\ell }_C(e_{-},e_{+})\forall e_{-},e_{+}\in {\mathbb{Z}}_{+}.$$

$${\hat{\ell }}_C(e_1,e_2)=\frac{e_1+e_2}{m-e_1+1},\forall e_1,e_2\in {\mathbb{R}}_{+}.$$

$$\frac{\partial \hat{\ell }}{\partial e_1}=\frac{m+e_2+1}{(m-e_1+1{)}^2},$$

$$\frac{{\partial }^2\hat{\ell }}{\partial e_1^2}=\frac{2(m+e_2+1)(m-e_1+1)}{(m-e_1+1{)}^4}\ge 0$$

$$\arg \underset{y_a,y_b}{\min }-⟨w,\varphi (x,y_a)⟩-\Delta (y^{\ast },y_b)\text{s.t. }{y}_a=y_b.$$

$$\mathcal{L}(y_a,y_b,\lambda )=$$

$$+{\lambda }^T(y_a-y_b)+\frac{\rho }{2}\Vert y_a-y_b{\Vert }_2^2$$

$$\min f(x)+g(x)$$

$$\text{s.t. }Ax+Bz=c$$

$$y_b=\arg \underset{y}{\max }\Delta (y^{\ast },y)+r(y)$$

$$r(A)=\sum _{j\in A}{w}^j$$

$$w^j=-\frac{\rho }{2}(y_a^j+y^{\ast j}+u^j{)}^2+\frac{\rho }{2}(y_a^j-y^{\ast j}+u^j{)}^2,\forall j\in V$$

$${\ell }_{\text{marginal}}(j)=\ell (j)-\ell (j-1);$$

$${\ell }_{\text{augmented}}(j)={\ell }_{\text{marginal}}(j)+w^{{\pi }^j};$$

$${\ell }_{\text{all}}(j)=\sum _{k=1}^j{\ell }_{\text{augmented}}(k);$$

$$r(A)=\sum _{j\in A}{w}_{\text{neg}}^j,\forall A\subseteq M,$$

Full text

1 Introduction

Discriminative structured prediction is a valuable tool in computer vision that has been applied to a wide range of application areas, and in particular object detection and segmentation [2, 6, 29, 31, 32, 39]. It is frequently applied using variants of the structured output support vector machine (SVM) [42, 43] in which a domain specific discrete loss function is upper bounded by a piecewise linear surrogate. In the case of image segmentation, this discrete loss function has frequently been taken to be the Hamming loss, which simply counts the number of incorrect pixels [2, 39]. Following the principle of empirical risk minimization, one might expect that minimization of the desired loss at training time would lead to the best performing loss at test time. However, it has recently been shown that in the finite sample regime, minimizing a different loss can lead to better performance even when measured using Hamming loss [31]. In that work, a supermodular loss function was employed, and a custom graph cuts solution was found to the loss augmented inference problem necessary for computation of a subgradient or cutting plane of the learning objective [19].

Several non-modular loss functions have been considered in the context of image segmentation, e.g. the intersection over union loss in the context of a Bayesian framework [28], an area/volume based label-count loss that enforces high-order statistics [32], or a layout-aware loss function that takes into account the topology/structure of the object [31]. A message passing based optimization scheme is proposed for optimizing several families of structured loss functions [40, 41], which assumes the loss function is constructed by a grammar for which the productions specify function composition [40]. By contrast, we provide a generic framework for decomposing the loss function from model inference that assumes a custom solver for the loss, but that does not assume the loss belongs to a specific compositional grammar. We concern ourselves primarily with supermodular loss functions in this work as they lead to provable polynomial time loss augmented inference problems (an essential step in training structured output SVMs), while non-supermodular loss functions lead to NP-hard optimization in general.

In general, it is a time consuming process to develop custom loss-augmented solvers for different combinations of loss functions and inference procedures. We show in this work a direct combination of two submodular graph cuts procedures may in fact lead to a non-submodular minimization problem, and reparametrizations or novel graph constructions may be necessary. Furthermore, if we attempt to solve a non-submodular minimization problem approximately, this may lead to poor convergence of the learning procedure and catastrophic failure of the learning algorithm [12].

An alternative approach is to resort to generic submodular optimization algorithms, such as that of Iwata [18] which has complexity $\mathcal{O}(n^{4}T+n^{5}\log M)$, or Orlin [30] with complexity $\mathcal{O}(n^{6}+n^{5}T)$, where $T$ is the time for a single function evaluation and $M$ is an upper bound on the absolute value of the function. Although these optimization algorithms are polynomial, the exponent is sufficiently large as to render them infeasible for images of even less than one megapixel. In practice, the Fujishige-Wolfe minimum norm algorithm [13, 14] is empirically faster [10]. However, we will show that even this state of the art optimization strategy is infeasible for relatively small consumer images.

Specific subclasses of submodular functions come with lower complexity optimization algorithms, and we should be able to exploit these known classes in a general learning framework. Examples include decomposable submodular functions [38, 27], several notions of symmetry [21, 33], and graph partition problems [22, 11]. A problem with the current API for loss augmented inference is that it is assumed that the loss function will decompose with a structure compatible to that of the inference problem. We address the case that this assumption does not hold and that separate efficient optimization procedures are available for the loss and for inference.

We propose to use Lagrangian splitting techniques to separate loss maximization from the inference problem. Strategies such as dual decomposition have become popular in Markov Random Field (MRF) inference [23], while later developments such as the alternating direction method of multipliers (ADMM) [4, 7] have improved convergence guarantees. Other strategies involving a quadratic penalty term have also been proposed in the literature, although still with the assumption that the loss decomposes as the inference [26]. We make use of ADMM to separate these inference problems and apply them to several supermodular loss functions that cannot be straightforwardly incorporated in a submodular graph partition problem for loss augmented inference. Instead we allow separate optimization strategies for the loss maximization and inference procedures yielding substantially improved computational performance, while making feasible the application of a wide range of supermodular loss functions by changing a single line of code.

This article is an extended version of [47] with additional theoretical contributions and experimental results. Specifically, we have added:

Section 2.2: a supermodular loss function, the square loss function; 2. 2.

Section 2.3: a new section on supermodular loss functions through biconvexity, with the definition of biconvexity, a proposition that biconvexity characterizes supermodularity for an important class of loss functions, and its proof; 3. 3.

Section 2.4: a novel supermodular loss function from biconvexity; 4. 4.

Section 2.6: a new section on ADMM convergence theorems; 5. 5.

Section 2.7: a new section on the optimization algorithm related to the novel supermodular loss functions in Sections 2.2 and 2.4; 6. 6.

Tables 1 and 2: additional results with more parameter values; 7. 7.

Section 3.2: new experimental results with the supermodular loss functions introduced in Sections 2.2 and 2.4; 8. 8.

Figures 8 to 10: additional qualitative segmentation results.

2 Methods

We discriminatively train a graph cuts based segmentation system using a structured SVM [43]. We first construct a supermodular loss function that is solvable with graph cuts, but that when incorporated in a joint loss-augmented inference leads to non-submodular potentials, which causes graph cuts based optimization to fail. We therefore use an ADMM based decomposition strategy to perform loss augmented inference. This strategy consists of alternatingly optimizing the loss function and performing maximum a posteriori (MAP) inference, with each process augmented by a quadratic term enforcing the labeling determined by each to converge to the optimum of the sum.

The structured output SVM is a discriminative learning framework that has been applied in diverse computer vision applications [2, 6, 29, 31, 32, 39]. Given a training set of labeled images $\{(x_{1},y^{*}_{1}),\dots,$ $(x_{n},y^{*}_{n})\}$ $\in\left(\mathcal{X}\times\mathcal{Y}\right)^{n}$, where $\mathcal{Y}=\{-1,1\}^{p}$ for a binary segmentation problem with $p$ pixels, it optimizes a regularized convex upper bound to a structured loss function, $\Delta:\mathcal{Y}\times\mathcal{Y}\to\mathbb{R}_{+}$. $\Delta$ measures the mismatch between a ground truth labeling and a hypothesized labeling. With $\Delta$ provided as an input, the structured SVM with margin rescaling minimizes [43]:

[TABLE]

In the case of image segmentation, we may interpret $\langle w,\phi(x,y)\rangle$ as a function that is monotonic in the log probability of the joint configuration of observed and unobserved variables $(x,y)$ as determined by a CRF [25]. Under this interpretation, a standard definition of $\phi$ is

[TABLE]

where $\phi_{u}$ determines a vector of features, a linear combination of which form the unary potentials of the CRF, and $\phi_{p}$ determines the pairwise potentials over a model specific edge set $\mathcal{E}$. In this work, we have set $\phi_{p}(x,\cdot,\cdot):\{-1,1\}^{2}\to\{0,1\}^{3}$ to map to an indicator vector of three cases: (i) $y^{k}=y^{l}=-1$, (ii) $y^{k}\neq y^{l}$, or (iii) $y^{k}=y^{l}=+1$, and have placed hard constraints on the corresponding entries of $w$ in the optimization of the structured SVM to ensure that the pairwise potentials in the energy minimization problem remain submodular [48].

During training of the structured SVM, we must perform loss augmented inference in order to compute a subgradient of the loss. In the case of margin rescaling, this consists of computing

[TABLE]

If $\mathcal{Y}$ is isomorphic to $\{-1,1\}^{p}$ for some $p$, $\Delta(y^{*},\cdot)$ will be isomorphic to a set function $\ell:\mathcal{P}(V)\to\mathbb{R}_{+}$ where $\mathcal{P}(V)$ is the power set of a base set with $|V|=p$. This allows us to discuss the properties of such loss functions $\Delta$ in terms of the language of set functions as occurs in real analysis [20] and discrete optimization [37]. In particular, we are interested in $\Delta$ corresponding to a supermodular set function $\ell$ [37, 44]:

Definition 1** (Supermodular set function [14]).**

A supermodular set function is a set function $\ell:\mathcal{P}(V)\to\mathbb{R}$ which satisfies: for every $A,B\subseteq V$ with $A\subseteq B$ and every $v\in V\setminus B$ we have that

[TABLE]

A function is submodular if its negative is supermodular. Given the definition of supermodularity, we may now define when a loss function $\Delta:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}_{+}$ is supermodular.

Definition 2** (Supermodular loss function [44]).**

A loss function $\Delta:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}_{+}$ is called supermodular if, for every $y\in\mathcal{Y}$, the unique set function $\ell$ such that $\Delta(y,\tilde{y})\mapsto\ell(\{j|y^{j}\neq\tilde{y}^{j}\})$ is supermodular.

We note that in Definition 2 the mapping to the set function $\ell$ has an explicit dependency on the ground truth labeling $y$ and varies per training image.

Necessary to the sequel of the article, we introduce also the definition of a symmetric set function:

Definition 3** (Symmetry).**

A set function $\ell:\mathcal{P}(V)\mapsto\mathbb{R}$ is symmetric if $\ell(A)=c(|A|)$ for some function $c:\mathbb{Z}_{+}\mapsto\mathbb{R}$. ($\mathbb{Z}$ is the set of integers and $\mathbb{Z}_{+}$ is the set of non-negative integers.)

Theorem 1** (Cardinality-based set function [3]).**

If $\ell:\mathcal{P}(V)\mapsto\mathbb{R}$ and there exist a function $c:\mathbb{Z}_{+}\mapsto\mathbb{R}$ such that $\ell(A)=c(|A|)$, where $|\cdot|$ is the cardinality of $A$. Then $\ell$ is supermodular if and only if $c$ is convex.

As we have guaranteed that maximization of $\langle w,\phi(x_{i},\tilde{y}_{i})\rangle$ with respect to $\tilde{y}$ corresponds to a submodular minimization problem, the loss augmented inference as in Equation (4) remains a submodular minimization when $\Delta$ is supermodular and can be aligned with the inference, and therefore polynomial time solvable. By contrast, non-supermodular $\Delta$ result in NP-hard optimization problems in general.

Modular loss functions, such as Hamming loss, can be incorporated into the unary potentials in a graph cuts optimization framework for loss augmented inference. However, the formulation of loss augmented inference with supermodular losses as a graph cuts problem is not straightforward, despite previous work (in which a custom graph cuts formulation was derived for a specific family of supermodular losses) that indicated a supermodular loss can lead to improved segmentation quality [31]. Moreover, while supermodular loss functions guarantee polynomial time solvability, they do not do so with low order polynomial guarantees in general. We have observed that the Fujishige-Wolfe algorithm is infeasible to apply even in the case of sub-megapixel images, and scales poorly for useful supermodular loss functions. Consequently, we develop a general framework for decomposing loss augmented inference based on ADMM. This framework solely relies on a loss function being able to be efficiently optimized in isolation using a specialized solver specific to the loss function.

2.1 A supermodular loss function for binary image segmentation

As a first example, we propose a loss function that is itself optimizable with graph cuts. The loss simply counts the number of incorrect pixels plus the number of pairs of neighboring pixels that both have incorrect labels

[TABLE]

where $[\cdot]$ is Iverson bracket notation, $\mathcal{E}_{\ell}$ is a loss specific edge set and $\gamma$ is a positive weight. We have used 8-connectivity for the loss function in the experiments (Figure 1(a)), referred to as “8-connected loss” in the sequel. We may identify this function with a set function to which the argument is the set of mispredicted pixels.

Proposition 1**.**

Maximization of the loss function in Equation (6) is equivalent to a supermodular function maximization problem.

Proof sketch.

Equation (6) is isomorphic to a binary random field model for which label is $1$ iff a pixel has a different label from the ground truth. Neighboring pixels that both have label $1$ contribute a positive amount to the energy, while all other configurations contribute zero. This corresponds to a supermodular function following Definition 1. ∎∎

This loss function emphasizes the importance of correctly predicting adjacent groups of pixels, e.g. those present in thin structures more than one pixel wide. While the pairwise potential in $\langle w,\phi(x,y)\rangle$ has a tendency to reduce the perimeter of the segment, the loss strongly encourages the correct identification of adjacent pixels. We will observe in the experimental results that the use of this loss function during training improves the test time prediction accuracy, even when measuring in terms of Hamming loss.

It may appear at first glance that the structure of this loss function is aligned with that of the inference, and that we can therefore jointly optimize the loss augmented inference with a single graph cuts procedure. Indeed, the loss function is isomorphic to a supermodular set function, and the inference is isomorphic to a supermodular set function, both of which can be solved by graph cuts. However, the isomorphisms are not the same. The loss function maps to a set function by considering the set of pixels that are incorrectly labeled, while the inference maps to a set function by considering the set of pixels that are labeled as foreground. Shown in Figure 1 is the pairwise potential for an edge with $y^{*k}=+1$ and $y^{*l}=-1$.

If we apply a single mapping, the inference procedure can be solved by graph cuts when the sum of the diagonal elements of $E$ is less than the sum of the off diagonal elements. While it is enforced during optimization that $w_{00}+w_{11}-w_{01}-w_{10}\geq 0$, the presence of $\gamma$ in the off diagonal, for which the exact position depends on the value of $y^{*}$, removes the guarantee of a resulting submodular minimization problem. We therefore consider a Lagrangian based splitting method to solve the loss augmented inference problem in Section 2.5

2.2 Symmetric supermodular loss function: square loss

As a second example, we consider the following loss which simply takes the square of the number of mis-predictions. This function is not readily incorporated in graph-cuts, as the square induces a pairwise dependency between all pixels.

[TABLE]

where $\alpha>0$ is a scale factor to prevent the value to be too large in an image scale problem. We used $\alpha=\sqrt{|y^{*}|}$ in our setting, where $|\cdot|$ is the number of positive labels of $y^{*}$. This is a function on the misprediction set which only depends on the size of the input set i.e. $\ell(A)=\left(\frac{|A|}{\alpha}\right)^{2}$. As the square function is a convex function, $\Delta_{\text{S}}$ is a supermodular loss w.r.t. the misprediction set. Then maximizing the loss itself is a supermodular maximization i.e. a submodular minimization problem, following Theorem 1.

2.3 Supermodular Loss Functions Through Biconvexity

In this section, we develop a family of supermodular loss functions based on biconvex functions of the number of false positives and false negatives.

Definition 4** (Biconvexity [16]).**

A function $f:\mathcal{A}\times\mathcal{B}\rightarrow\mathbb{R}$ is called a biconvex function if

[TABLE]

is a convex function on $\mathcal{B}$ for every fixed $a\in\mathcal{A}$ and

[TABLE]

is a convex function on $\mathcal{A}$ for every fixed $b\in\mathcal{B}$.

• Remark

The usual definition of biconvexity specifies that the function be defined over a biconvex set [16, Definition 1.1], but for the purpose of this section we will restrict ourselves to $\mathcal{A}$ and $\mathcal{B}$ being convex sets so that biconvexity of the domain of $f$ follows trivially.

Denote by $m$ the number of positive labels in the ground truth labeling $y^{*}$,

[TABLE]

Proposition 2** (Biconvexity characterizes supermodularity).**

For a given ground truth labeling $y^{*}$ and a given prediction $\tilde{y}$, let $e_{-}$ denote the number of false negatives, and $e_{+}$ denote the number of false positives:

[TABLE]

where $[\cdot]$ is Iverson bracket notation. The following holds

[TABLE]

is a supermodular loss function iff $\exists\hat{\ell}:[0,m]\times[0,p-m]\mapsto\mathbb{R}_{+}$ that is a biconvex function and

[TABLE]

In particular, we may select $\hat{\ell}$ to be the convex closure of $\ell$ [3, Section 5.1].

Proof.

We first show that supermodularity implies the existence of a corresponding biconvex function. [3, Proposition B.2] indicates that for a function to be supermodular, all contractions of that function must be supermodular, in particular the contractions achieved by fixing a set of false positives and fixing a set of false negatives.

For the contraction obtained by fixing false positives to be supermodular, we have from Theorem 1 that there exists a convex function that specifies the contraction. Similarly for the contraction obtained by fixing false negatives to be supermodular, there exists a (different) convex function that specifies the contraction. Combining all such contractions and convex functions yields the conditions in Definition 4 for integral points. The existence of a function satisfying these conditions for non-integral points is obtained by noting that the convex closure satisfies the required properties.

It now remains to show that a biconvex function yields a supermodular set function. Given a biconvex function of the number of false negatives and the number of false positives, by Definition 4, the function obtained by fixing the number of false positives is convex in the number of false negatives, then we have from Theorem 1 that the set function restricted to the set of foreground pixels is supermodular. A symmetric argument gives that the restriction to the set of background pixels is also supermodular. ∎∎

Several popular loss functions such as Intersection over Union loss [45, Equation (43)] or Sørensen-Dice loss [46, Definition 11] can be specified as functions of the number of false positives and false negatives, but both have been shown to be non-supermodular. In the next section we will develop a novel supermodular loss function by specifying an increasing biconvex function of $e_{-}$ and $e_{+}$.

2.4 A novel loss function from biconvexity

The Intersection over Union loss [6, Equation (7)] has been shown to be non-supermodular [45, Proposition 10]:

[TABLE]

We develop here a novel loss function that is similar in flavor to Equation (15) but we will see that it is supermodular:

[TABLE]

We can verify that $0\leq\Delta(y^{*},\tilde{y})$, $\forall\tilde{y}\in\mathcal{Y}$, and $\Delta(y^{*},y^{*})=0$.

Given a ground truth labeling $y^{*}$, we can consider $m$ to be a constant. With the notation in Equation (11) and Equation (12), we can write $\Delta_{C}$ as a function of $e_{-}$ and $e_{+}$, denoted $\ell_{C}$:

[TABLE]

Proposition 3**.**

There exits a function $\hat{\ell}_{C}:\mathbb{R}_{+}\times\mathbb{R}_{+}\mapsto\mathbb{R}_{+}$ that is biconvex and

[TABLE]

Proof.

We set

[TABLE]

It is straightforward that it satisfies Equation (14) in Proposition 2. We now prove that this is a biconvex function. We note that with $e_{1}$ fixed, $\hat{\ell}_{C}$ is linear in $e_{2}$ and therefore $\hat{\ell}_{C}(e_{1},\cdot)$ is convex.

Now to show that $\hat{\ell}_{C}(\cdot,e_{2})$ is convex, we calculate its first and second derivatives with respect to $e_{1}$:

[TABLE]

Given the fact that $e_{1}$ is the number of false negatives and $m$ is the number of ground truth positive labels, we have $e_{1}\leq m$. All parenthesized terms of Equation (21) must therefore be strictly positive. As $\hat{\ell}_{C}(\cdot,e_{2})$ is twice differentiable everywhere and its second derivative is non-negative, $\hat{\ell}_{c}$ is convex wrt $e_{1}$. ∎∎

Following Proposition 2, we then have the following corollary:

Corollary 1**.**

$\Delta_{C}$* in Equation (16) is supermodular.*

2.5 ADMM algorithm for loss augmented inference

Several Lagrangian based decomposition frameworks have been proposed, such as dual decomposition and ADMM [7], with the latter having improved convergence guarantees. We have also observed a substantial improvement in performance using ADMM over dual decomposition in our own experiments. Here we consider a splitting method to optimize the minimization of the negative of Equation (4), which is equivalent to finding the most violated constraint in cutting plane optimization:

[TABLE]

and we form the augmented Lagrangian as

[TABLE]

where $\rho>0$. (23) can be optimized in an iterative fashion by Algorithm 1 [7].

The saddle point of the Lagrangian will correspond to an optimal solution over a convex domain, while we are optimizing w.r.t. binary variables. Strictly speaking, we may therefore consider the linear programming (LP) relaxation of our loss augmented inference problem, followed by a rounding post-processing step. We use a standard stopping criterion as in [7]: the primal and dual residuals must be small with an absolute criterion $\epsilon^{\text{abs}}=10^{-4}$ and a relative criterion $\epsilon^{\text{rel}}=10^{-2}$. In practice, we have found that discretizing the quadratic terms and incorporating them into the unary potentials of the respective graph cuts problems is more computationally efficient, while yielding results that are nearly identical with exact optimization with a primal-dual gap of 0.01%. We show in the experimental results that this strategy yields results almost identical to those of an LP relaxation.

In general, we simply need task-specific solvers for Line 3 and Line 4 of Algorithm 1. These solvers need not use a single graph cut algorithm, and can therefore exploit any available structure even though it may not be present, or aligned, between the two subproblems. Although we have used this framework for the specific supermodular loss functions described in the previous subsection, we note that this provides an API for the structured output SVM framework alternate to that provided by SVMstruct [43]. We have released our structured prediction toolbox as an open source project, enabling the application of this strategy to diverse structured prediction problems with non-modular loss functions.

2.6 ADMM Convergence

Consider the standard form of the problem solved by ADMM:

[TABLE]

with variables $x\in\mathbb{R}^{n}$ and $z\in\mathbb{R}^{m}$, where $A\in\mathbb{R}^{p\times n}$, $B\in\mathbb{R}^{p\times m}$, and $c\in\mathbb{R}^{p}$. Following [7], some general convergence results for ADMM are considered in this section.

Assumption 1**.**

The (extended-real-valued) functions $f:\mathbb{R}^{n}\mapsto\mathbb{R}\cup\{+\infty\}$ and $g:\mathbb{R}^{m}\mapsto\mathbb{R}\cup\{+\infty\}$ are closed, proper, and convex.

Assumption 2**.**

The unaugmented Lagrangian of the problem has a saddle point.

If Assumption 1 and Assumption 2 hold, the ADMM algorithm guarantees: (1) the residual convergence: $r^{k}\to 0$ as $k\to\infty$, i.e., the iterates approach feasibility; (2) the objective convergence: $f(x^{k})+g(z^{k})\to p$ as $k\to\infty$, i.e., the objective function of the iterates approaches the optimal value; (3) and the dual variable convergence: $y^{k}\to y$ as $k\to\infty$, where $y$ is a dual optimal point. Proofs of the residual and objective convergence results are given in [7].

2.7 Optimization

As shown in Line 4 in Algorithm 1, we need to solve the subproblem that minimizes the negative of the loss function augmented by a term from the ADMM iteration. It is equivalent to maximizing the sum of the loss function and the negative of the ADMM term. Among the three examples of supermodular loss functions we proposed, maximizing the 8-connected loss in Equation (6), augmented by a modular term from the ADMM iteration, can be solved by a modified graph-cut. Maximizing the square loss (Equation (7)) and the biconvex loss (Equation (16)) can also be solved efficiently, as we will show in this section.

Explicitly, we maximize over the sum of a supermodular loss function and a modular function:

[TABLE]

where $r(y)=-\frac{\rho}{2}(\|y_{a}-y+u\|_{2}^{2})$ is an asymmetric modular function wrt the misprediction set $\{j|y^{*j}\neq y^{j}\}$ for a given $y_{a}$ and $u$ at the current iteration (we discard the supercript $t$ for simplicity). We know that any modular function can be written as

[TABLE]

for some coefficient vector $w\in\mathbb{R}^{|V|}$. In our case, we have

[TABLE]

Under the assumption that $\Delta(y^{*},y_{b})$ is a symmetric loss function, such as the square loss in Equation (7), Algorithm 2 solves the required optimization efficiently.

By exploiting the symmetry properties of the loss function, all operations in Algorithm 2 are linear except for the first sorting operation.

For $\Delta(y^{*},y_{b})$ biconvex, as in Equation (16), we analyze the problem wrt to false positives and false negatives separately. For a given ground truth labeling $y^{*}$, we note that the subset $M:=\{j|y^{*j}=+1\}$ is the set of all possible false negatives. $|M|=m$ following Equation (10), $\{j|y^{*j}=-1\}$ is the set of all possible false positives, i.e. $V\setminus M$. We first rewrite the modular function as a coefficient vector of ground truth positive entries and ground truth negative entries separately,

[TABLE]

for two coefficient vectors $w_{\text{neg}},w_{\text{pos}}\in\mathbb{R}^{|V|}$.

Under the assumption that $\Delta(y^{*},y_{b})$ is a biconvex function, as in Equation (16), Algorithm 3 is an efficient solver for the resulting optimization.

3 Experimental Results

In this section, we consider a foreground/background segmentation task. We compare the prediction using our proposed supermodular loss functions with the prediction using Hamming loss. We show that: (i) our proposed splitting strategy is orders of magnitude faster than the minimum norm point algorithm; (ii) our strategy yields results nearly identical to a LP-relaxation while being much faster in practice; and (iii) training with the same supermodular loss as during test time yields better performance.

Datasets

The dataset provided by [17, 5] contains color images in RGB space, ground truth foreground/background segmentations, and user-labelled seeds (see Figure 2, Figure 2, and Figure 2, respectively). As we are discriminatively training a class specific segmentation system in our experiments, we focus on the images in which the foreground objects are people. We compute in total 18 unary features following [31]. Figure 2 to Figure 2 show examples of the extracted features.

IBSR Dataset

We additionally utilise the Internet Brain Segmentation Repository (IBSR) dataset [34], which consists of T1-weighted MR images. Images and masks have been linearly registered and cropped to $145\times 158\times 123$. We choose one horizontal slice within each volume and we follow the feature extraction procedure as in [1].

3.1 Training with the 8-connected loss function

We use the ADMM splitting strategy to solve the minimization problem in Equation (22). We use the GCMex - MATLAB wrapper for the Boykov-Kolmogorov graph cuts algorithm [15, 8, 9, 22] to solve the optimization problems on Line 3 for the inference. Results computed with different values of $\gamma>0$ are shown in Table 1 and Table 3. During the training stage, we use $\rho=0.1$ for the ADMM step-size parameter. The regularization parameter $C$ in Equation (1) is chosen by cross-validation in the range $\{10^{i}|-2\leq i\leq 2\}$. We additionally train and test with Hamming loss as a comparison.

At test time, we have computed the unnormalized Hamming loss, the intersection over union loss (IoU), and our 8-connected loss for each training scenario. We have performed several random train-test splits in order to compute error bars on the loss estimates. During testing stage, we evaluate one prediction as the average loss value for all images in the testing set. We compare different loss functions during training and during testing and measure the empirical loss values. We randomly split the data into training and testing sets five times to obtain an estimate of the average performance.

Empirical Results

We show in Table 1 and Table 3 the empirical error values by training with the 8-connected supermodular loss compared with training with the Hamming loss (labeled 0-1). In Table 1 we show the results by using different values of $\gamma$ for the 8-connected loss. We notice that in all cases, training with the same supermodular loss as used for testing has achieved the best performance, i.e. lower error values. Training with the supermodular loss even outperforms training with Hamming loss when measured by Hamming loss on the test set. Wilcoxon sign rank tests are shown in Table 2, which shows that training with the supermodular loss functions gives significantly better results in nearly all cases.

We have additionally tried training with a joint graph cuts loss augmented inference using the pairwise potentials illustrated in Figure 1. However, due to the non-submodular potentials, the graph cuts procedure does not correctly minimize the energy resulting in incorrect cutting planes that causes optimization to fail after a small number of iterations. The performance of this system was effectively random, and we have not included these values in Table 1.

Qualitative segmentation results are shown in Figure 3. In Figure 6 and 7 we show a pixelwise comparison of the predictions. The 8-connected loss achieves better performance on the foreground/background boundary, as well as on elongated structures of the foreground object, such as the head and legs, especially when the appearance of the foreground is similar to the background.

We also ran a baseline comparing non-submodular loss augmented inference with the QPBO approach [35]. We computed pairwise energies as in Figure 1(a). QPBO found loss augmented energies across the dataset of $1.1\times 10^{6}\pm 3\times 10^{5}$ while ADMM found loss augmented energies of $3.7\times 10^{6}\pm 8\times 10^{5}$, a substantial improvement.

3.2 Training with the square loss and the biconvex loss

We show in Table 4 the empirical error values by training with the square loss (labeled $\Delta_{S}$), and with the biconvex loss (labled $\Delta_{C}$), compared to training with the Hamming loss (labeled 0-1). We can see that training with the same supermodular loss during test time yields better performance than training with the Hamming loss, which validates the correctness of the ADMM splitting strategy with more loss/inference combinations.

Qualitative segmentation results are shown in Figure 8. Pixelwise comparison of the segmentation results using the square loss and the biconvex loss are shown in Figure 9 and Figure 10, respectively.

3.3 Computation Time

In addition, when using the 8-connected loss, we compare the time of one calculation of the loss augmented inference by the ADMM algorithm and by the minimum norm point algorithm [14] (MinNorm). For MinNorm, we use the implementation provided in the SFO toolbox [24]. Although it has been proven that in $t$ iterations, the MinNorm returns an $O(1/t)$-approximate solution [10], the first step of this algorithm is to find a point in the submodular polytope, which alone is computationally intractable even for small $600\times 400$ pixel images. Therefore, we measure the computation time on downsampled images, showing the growth in computation as a function of image size (Figure 4 and Figure 5). The running times are recorded on a machine with a 3.20GHz CPU. Similarly, a dual-decomposition baseline took orders of magnitude longer computation than the ADMM approach, following known convergence results [7].

We measure the computation time for 120 calculations of the loss augmented inference by ADMM and MinNorm on different sized images. From Figure 4 and Figure 5 we can see that ADMM is always faster than the MinNorm by a substantial margin, and around 100 times faster when the problem size reaches $10^{3}$. The computing time for both ADMM and MinNorm vary approximately linearly in log-log scale, while MinNorm has a higher slope, suggesting a worse big-$\mathcal{O}$ computational complexity. We note that theoretical bounds on MinNorm are currently weak and the exact complexity is unknown [10].

Although it is immediately clear from Figure 5 that ADMM is substantially faster than the minimum norm point algorithm, we have performed Wilcoxon sign rank tests that show this difference is significant with $p<10^{-20}$ in all settings.

3.4 Comparison to LP-relaxation

We additionally compare ADMM to an LP relaxation procedure for the loss augmented inference to determine the accuracy of our optimization in practice, with using the 8-connected loss function and the Hamming loss (0-1). For the implementation of the LP relaxation, we use the UGM toolbox [36]. We show in Table 5 the comparison between using ADMM and the LP relaxation. The first column represents the energy achieved by the loss augmented inference (Equation (4)). We observe that the (maximal) energy achieved by ADMM is almost the same as the LP relaxation: a difference of $0.4\%$. Columns 2–4 show the computing time for one calculation of the loss augmented inference on the downsampled images. Using an LP relaxation, the computation time is orders of magnitude slower, growing as a function of the image size. ADMM provides a more efficient strategy without loss of performance.

4 Discussion and Conclusion

A somewhat surprising result in Table 1 is that training with the supermodular loss results in better performance as measured by Hamming loss. This has been previously observed with a different loss function by [32, 31], and indicates that in the finite sample regime a supermodular likelihood can result in better generalization performance. This holds, although the model space and regularizer were identical in both training settings. We have observed the same effect with the other two supermodular loss functions, $\Delta_{S}$ and $\Delta_{C}$, indicating that this may be a broader property of supermodular loss functions.

Our results in terms of computation time give clear evidence for the superiority of ADMM inference when a specialized optimization procedure is available for the loss function. As shown in Figure 5, the Fujishige-Wolfe minimum norm point algorithm does not scale to typical consumer images (i.e. several megapixels), which indicates that loss functions for which a specialized optimization procedure is not available are likely infeasible for pixel level image segmentation without unprecedented improvements in general submodular minimization. Figure 5 shows that the log-log slope of the runtime for the min-norm point algorithm is higher than for ADMM, suggesting a worse computational compexity. One may wish to employ the result that early termination of the min-norm point algorithm gives a guaranteed approximation of the exact result, but even this is infeasible for images of the size considered here. In addition, Table 5 suggests that ADMM provides a more efficient strategy without lost of performance compared to using an LP-relaxation. Joint graph-cuts optimization for loss augmented inference results in non-submodular pairwise potentials and graph-cuts fails to correctly minimize the joint energy. As a result, a cutting plane optimization of the structured output SVM objective fails catastrophically, and the resulting accuracy is on par with a random weight vector. Consequently, the ADMM technique yielded the only feasible training strategy.

In this work, we propose three novel supermodular loss functions. We have shown that using supermodular loss functions achieves improved performance both in qualitative and quantitative terms on a binary segmentation task. We observe that a key advantage of the proposed supermodular losses over modular losses, e.g. Hamming loss, is an improved ability to find elongated regions such as heads and legs, or thin articulated structures in medical images.

Previous to our work, specialized inference procedures had to be developed for every model/loss pair, a time consuming process. By contrast, we have proposed a Lagrangian splitting technique based on ADMM to perform general loss augmented inference. We demonstrate the feasibility of the ADMM algorithm for loss augmented inference on an interactive foreground/background segmentation task, for which alternate strategies such as the Fujishige-Wolfe minimum norm point algorithm are infeasible. Our proposed ADMM algorithm provides a strategy to solve the loss augmented inference as two separate subproblems. This provides an alternate API for the structured output SVM framework to that of SVMstruct [43]. We envision that this can be of use in a wide range of application settings, and an open source general purpose toolbox for this efficient segmentation framework with supermodular losses is available for download from https://github.com/yjq8812/efficientSegmentation.