Boosting with Structural Sparsity: A Differential Inclusion Approach

TL;DR

This paper introduces Split LBI, a novel boosting algorithm based on differential inclusions that enhances structural sparsity control, outperforming generalized Lasso in theory and experiments across various applications.

Contribution

The paper proposes Split LBI, a new boosting algorithm leveraging differential inclusions for structural sparsity, with theoretical guarantees and practical advantages over existing methods.

Findings

Split LBI outperforms generalized Lasso in theory and experiments.

It achieves model selection consistency under weaker conditions.

The algorithm is effective in applications like image denoising and ranking.

Abstract

Boosting as gradient descent algorithms is one popular method in machine learning. In this paper a novel Boosting-type algorithm is proposed based on restricted gradient descent with structural sparsity control whose underlying dynamics are governed by differential inclusions. In particular, we present an iterative regularization path with structural sparsity where the parameter is sparse under some linear transforms, based on variable splitting and the Linearized Bregman Iteration. Hence it is called \emph{Split LBI}. Despite its simplicity, Split LBI outperforms the popular generalized Lasso in both theory and experiments. A theory of path consistency is presented that equipped with a proper early stopping, Split LBI may achieve model selection consistency under a family of Irrepresentable Conditions which can be weaker than the necessary and sufficient condition for generalized Lasso. Furthermore, some $\ell_2$ error bounds are also given at the minimax optimal rates. The utility and benefit of the algorithm are illustrated by several applications including image denoising, partial order ranking of sport teams, and world university grouping with crowdsourced ranking data.

Figures

Tables

Table 1. Table 1: Mean AUC (with standard deviation) comparisons where Split LBI 1.4 beats genlasso . The first is for the standard Lasso, and the second is for the 1-D fused Lasso in Example   1 .

genlasso	Split LBI
	$\nu =1$	$\nu =5$	$\nu =10$
$.9426$	$.9845$	$.9969$	$.9982$
$\left(.0390\right)$	$\left(.0185\right)$	$\left(.0065\right)$	$\left(.0043\right)$

Table 2. Table 2: Universities in the 1st group.

Harvard University	Princeton University
Stanford University	University of California, Berkeley
Yale University	Cornell University
University of California, Los Angeles	University of Cambridge (UK)
California Institute of Technology	University of Oxford (UK)
Columbia University	University of Pennsylvania
Carnegie Mellon University	University of California, San Diego
University of Michigan	New York University
Johns Hopkins University

Table 3. Table 3: The Universities in the 2nd Group

Massachusetts Institute of Technology (MIT)	University of Southern California
University of British Columbia (Canada)	University of Wisconsin-Madison
Peking University (China)	Northwestern University
University of Chicago	Swiss Federal Institute of Technology (Switzerland)
Brown University	Georgia Institute of Technology
Imperial College London (UK)	University of Washington
University of Toronto (Canada)	University of California, Santa Barbara
Duke University	University of Tokyo (Japan)
The University of Hong Kong (Hong Kong)	Purdue University
University of Texas at Austin	Dartmouth College
University of California, Irvine	University of California, Santa Cruz
University of California, Davis	Tsinghua University (China)
University of Maryland, College Park	London School of Economics and Political Science (UK)
Boston University	Hong Kong University of Science and Technology (Hong Kong)
University College London (UK)	Rice University

Equations

$$y=X{\beta }^{\star }+ϵ,{\gamma }^{\star }=D{\beta }^{\star }\left(S=\mathrm{supp}\left({\gamma }^{\star }\right),s=|S|\right),$$

$$\arg \underset{\beta }{\min }\left(\frac{1}{2n}{\Vert y-X\beta \Vert }_2^2+\lambda {\Vert D\beta \Vert }_1\right).$$

$$\ell \left(\beta ,\gamma \right):=\frac{1}{2n}{\Vert y-X\beta \Vert }_2^2+\frac{1}{2\nu }{\Vert \gamma -D\beta \Vert }_2^2(\nu >0).$$

$${\beta }_{k+1}$$

$$z_{k+1}$$

$${\gamma }_{k+1}$$

$$\mathcal{S}\left(z,\lambda \right)=\mathrm{sign}(z)\cdot \max \left(|z|-\lambda ,0\right)(\lambda \ge 0).$$

$$\rho \in \partial {\Vert \gamma \Vert }_1,z=\rho +\gamma /\kappa$$

$${\beta }_{k+1}/\kappa$$

$${\rho }_{k+1}+{\gamma }_{k+1}/\kappa$$

$${\rho }_k$$

$$\dot{\beta }(t)/\kappa$$

$$\dot{\rho }(t)+\dot{\gamma }(t)/\kappa$$

$$\rho (t)$$

$$0$$

$$\dot{\rho }(t)$$

$$\rho (t)$$

$$\dot{\rho }(t)$$

$$\rho (t)$$

$$\Sigma =\Sigma (\nu ):=\left(I-D{A}^{†}{D}^T\right)/\nu ,\text{and}A=A(\nu )=\nu X^{\ast }X+D^{T}D.$$

$$\beta (t)=\arg \underset{\beta }{\min }\ell \left(\beta ,\gamma (t)\right)=A^{†}\left(\nu X^{\ast }y+D^T\gamma (t)\right),$$

$$\dot{\rho }(t)$$

$$\rho (t)$$

$$\mathcal{L}:=(\ker (X)\cap \ker (D){)}^{\perp }=\mathrm{Im}\left(X^T\right)+\mathrm{Im}\left(D^T\right),$$

$$\mathcal{M}:=\left\{\beta :D_{S^c}\beta =0\right\}.$$

$$H=H(\nu ):={\nabla }^2\ell \left(\beta ,\gamma \right)\equiv \left(\begin{array}{cc}{X}^{\ast }X+D^{T}D/\nu & -D^T/\nu \\ -D/\nu & I_m/\nu \end{array}\right)$$

$$\min \left({\lambda }_{\min ,+}\left(D\right),{\lambda }_{\min ,+}\left(D_{S^c}\right)\right)\ge {\lambda }_D,$$

$${\Lambda }_{\max }\left(D\right)\le {\Lambda }_D,$$

$${\Lambda }_{\max }\left(X^{\ast }X\right)\le {\Lambda }_X^2.$$

$${\beta }^T{X}^{\ast }X\beta \ge \lambda {\Vert \beta \Vert }_2^2,\text{for any}\beta \in \mathcal{L}\cap \mathcal{M}.$$

$$\left({\beta }^T,{\gamma }_S^T\right)\cdot H_{(\beta ,S),(\beta ,S)}(\nu )\cdot \left(\begin{array}{c}\beta \\ {\gamma }_S\end{array}\right)\ge \frac{C}{1+\nu }{\Vert \left(\begin{array}{c}\beta \\ {\gamma }_S\end{array}\right)\Vert }_2^2\left(\beta \in \mathcal{L},{\gamma }_S\in {\mathbb{R}}^s\right).$$

$$\underset{\rho \in [-1,1{]}^s}{\sup }{\Vert H_{S^c,(\beta ,S)}(\nu )H_{(\beta ,S),(\beta ,S)}(\nu {)}^{†}\cdot \left(\begin{array}{c}{0}_p\\ \rho \end{array}\right)\Vert }_{\infty }\le 1-\eta .$$

$${\Vert H_{S^c,(\beta ,S)}(\nu )H_{(\beta ,S),(\beta ,S)}(\nu {)}^{†}\cdot {\rho }_{(\beta ,S)}^{\star }\Vert }_{\infty }\le 1-\eta ,\text{where}{\rho }_{(\beta ,S)}^{\star }=\left(\begin{array}{c}{0}_p\\ {\rho }_S^{\star }\end{array}\right).$$

$${\Sigma }_{S,S}(\nu )⪰\frac{C^{\prime }}{1+\nu }I.$$

$${\Vert {\Sigma }_{S^c,S}(\nu ){\Sigma }_{S,S}(\nu {)}^{-1}\Vert }_{\infty }\le 1-\eta .$$

$$\mathrm{irr}(0):=\underset{\nu \to 0}{\lim }\mathrm{irr}(\nu ),\mathrm{irr}(\infty ):=\underset{\nu \to +\infty }{\lim }\mathrm{irr}(\nu ).$$

Full text

1 Introduction

In this paper, consider the recovery from linear noisy measurements of $\beta^{\star}\in\mathbb{R}^{p}$, which satisfies the following structural sparsity that the linear transformation $\gamma^{\star}:=D\beta^{\star}$ for some $D\in\mathbb{R}^{m\times p}$ has most of its elements being zeros. For a design matrix $X\in\mathbb{R}^{n\times p}$, let

[TABLE]

where $\epsilon\in\mathbb{R}^{n}$ has independent identically distributed components, each of which has a sub-Gaussian distribution with parameter $\sigma^{2}$ ($\mathbb{E}[\exp(t\epsilon_{i})]\leq\exp(\sigma^{2}t^{2}/2)$). In literature the linear transform $D$ has various examples including the Fourier transform, the wavelet transform, or graph gradient operators etc. Here $\gamma^{\star}$ is sparse, i.e. $s\ll m$. Given $(y,X,D)$, the purpose is to estimate $\beta^{\star}$ as well as $\gamma^{\star}$, and in particular, recovers the support of $\gamma^{\star}$.

There is a large literature on this problem. Perhaps the most popular approach is the following $\ell_{1}$-penalized convex optimization problem,

[TABLE]

Such a problem can be at least traced back to Rudin et al. (1992) as a total variation regularization for image denoising in applied mathematics; in statistics it is formally proposed by Tibshirani et al. (2005) as fused Lasso. As $D=I$ it reduces to the well-known Lasso (Tibshirani, 1996) and different choices of $D$ include many special cases, it is often called generalized Lasso (Tibshirani and Taylor, 2011) in statistics.

Various algorithms are studied for solving 1.2 at fixed values of the tuning parameter $\lambda$, most of which is based on the ADMM or Split Bregman using operator splitting ideas (see for examples Goldstein and Osher (2009); Ye and Xie (2011); Wahlberg et al. (2012); Ramdas and Tibshirani (2014); Zhu (2017) and references therein). To avoid the difficulty in dealing with the structural sparsity in $\|D\beta\|_{1}$, these algorithms exploit an augmented variable $\gamma$ to enforce sparsity while keeping it close to $D\beta$.

On the other hand, regularization paths are crucial for model selection by computing estimators as functions of regularization parameters. For example, Efron et al. (2004) studies the regularization path of standard Lasso with $D=I$, the algorithm in Hoefling (2010) computes the regularization path of fused Lasso, and the dual path algorithm in Tibshirani and Taylor (2011) can deal with generalized Lasso. Recently, Arnold and Tibshirani (2016) discussed various efficient implementations of the the algorithm in Tibshirani and Taylor (2011), and the related R package genlasso can be found in CRAN repository. All of these are based on homotopy method of solving convex optimization 1.2.

Our departure here, instead of solving 1.2, is to look at an extremely simple yet novel iterative scheme which finds a new regularization path with structural sparsity. We are going to show that it works in a better way than genlasso, in both theory and experiments.

1.1 New Algorithm: Split LBI

Define a loss function which splits $D\beta$ and $\gamma$,

[TABLE]

Now consider the following iterative algorithm,

[TABLE]

where the initial choice $z_{0}=\gamma_{0}=0\in\mathbb{R}^{m}$, $\beta_{0}=0\in\mathbb{R}^{p}$, parameters $\kappa>0,\ \alpha>0,\ \nu>0$, and the proximal map associated with a convex function $h$ is defined by $\mathrm{prox}_{h}(z)=\arg\min_{x}\|z-x\|^{2}/2+h(x)$, which is reduced to the shrinkage operator when $h$ is taken to be the $\ell_{1}$-norm, $\mathrm{prox}_{\|\cdot\|_{1}}(z)=\mathcal{S}\left(z,1\right)$ where

[TABLE]

The algorithm generates a sequence $(\beta_{k},\gamma_{k})_{k\in\mathbb{N}}$ which defines a discrete regularization path. Iteration 1.4a has appeared as $L_{2}$-Boost (Bühlmann and Yu, 2002) in machine learning and can be traced back to the Landweber Iteration in inverse problems (Yao et al., 2007) where early stopping regularization is needed against overfitting noise. On the other hand, 1.4b and 1.4c, generating a sparse regularization path on $\gamma_{k}$, is known as the Linearized Bregman Iteration (LBI) firstly proposed in Yin et al. (2008). Recently in sparse linear regression, Osher et al. (2016) shows that under nearly the same conditions as standard Lasso, LBI with early stopping may achieve sign consistency but with a less biased estimator than Lasso, and its limit dynamics will reach the bias-free oracle estimator which is optimal over all estimators. Equipped with a variable splitting between $D\beta$ and $\gamma$, algorithm 1.4 thus combines the $L_{2}$-Boost of $\beta$ for prediction and LBI of $\gamma$ for sparse structure. Hence in this paper we call 1.4 the Split LBI or Boosting with structural sparsity.

The gap $\|\gamma-D\beta\|_{2}^{2}/\nu$ controls the affinity between $D\beta$ and $\gamma$. As $\nu\to 0$, $D\beta=\gamma$ which meets the generalized Lasso constraint; while for a finite $\nu>0$, $D\beta$ is not necessarily sparse. Such an increase in degree of freedom, however, leaves us a new space for improving the model selection consistency, as we shall see in the following experiment and in later part of this paper for a theoretical development.

1.2 Improved Model Selection in Experiments

The following example shows that the iterative regularization path 1.4 can be more accurate than the regularization path of generalized Lasso, in terms of Area Under the Curve (AUC)111The “Area Under the Curve” is the area under the Receiver Operating Characteristic (ROC) Curve, whose definition can be seen for example in (Brown and Davis, 2006). measurement of the order of parameters becoming nonzero in consistent with the ground truth sparsity pattern (higher value of AUC means better performance of variable selection of an algorithm such that true parameters becoming nonzero along the algorithmic regularization path earlier than the null parameters). The following simple experiment illustrates such phenomena by simulations.

Example 1**.**

Consider two problems: standard Lasso and 1-D fused Lasso. In both cases, set $n=p=50$, and generate $X\in\mathbb{R}^{n\times p}$ denoting $n$ i.i.d. samples from $N(0,I_{p})$, $\epsilon\sim N(0,I_{n})$, $y=X\beta^{\star}+\epsilon$. $\beta_{j}^{\star}=2$ (if $1\leq j\leq 10$), $-2$ (if $11\leq j\leq 15$), and [math] (otherwise). For Lasso we choose $D=I$, and for 1-D fused Lasso we choose $D=(D_{1};D_{2})\in\mathbb{R}^{(p-1+p)\times p}$ such that $(D_{1}\beta)_{j}=\beta_{j}-\beta_{j+1}$ (for $1\leq j\leq p-1$) and $D_{2}=I_{p}$. Figure 1 shows the regularization paths by genlasso ($\{D\beta_{\lambda}\}$) and by iteration 1.4 (linear interpolation of $\{\gamma_{k}\}$) with $\kappa=200$ and $\nu\in\{1,5,10\}$, respectively. The generalized Lasso path is in fact piecewise linear with respect to $\lambda$ while we show it along $t=1/\lambda$ for a comparison. Note that the iterative paths exhibit a variety of different shapes depending on the choice of $\nu$. However, in terms of order of those curves entering into nonzero range, these iterative paths exhibit a better accuracy than genlasso. Table 1 shows this by the mean AUC of $100$ independent experiments in each case, where the increase of $\nu$ improves the model selection accuracy of Split LBI paths and beats that of generalized Lasso.

Why does Split LBI perform better in model selection than generalized Lasso? Some limit dynamics of algorithm 1.4 actually shed light on the cause.

1.3 Limit Differential Inclusions of Split LBI

Below we are going to derive several limit dynamics of Split LBI, which are differential inclusions and lead to explanations on how our algorithm might improve over generalized Lasso.

First of all, noting by the following Moreau Decomposition

[TABLE]

the Split LBI (1.4) can be rewritten as,

[TABLE]

where $\rho_{0}=\gamma_{0}=0\in\mathbb{R}^{m}$, $\beta_{0}=0\in\mathbb{R}^{p}$.

Now taking $\rho(k\alpha)=\rho_{k}$, $\gamma(k\alpha)=\gamma_{k}$, $\beta(k\alpha)=\beta_{k}$, and $\alpha\to 0$, 1.6 is a forward Euler discretization of the following limit dynamics, called Split Linearized Bregman Inverse Scale Space (Split LBISS) here.

Definition 1** (Split LBISS).**

For $\alpha\to 0$, define the following differential inclusion as the limit dynamics of Split LBI,

[TABLE]

where $\rho(t),\beta(t),\gamma(t)$ are right continuously differentiable, with $\dot{\rho}(t),\dot{\beta}(t),\dot{\gamma}(t)$ denoting the right derivatives in $t$ of $\rho(t),\beta(t),\gamma(t)$ respectively, and $\rho(0)=\gamma(0)=0\in\mathbb{R}^{m}$, $\beta(0)=0\in\mathbb{R}^{p}$.

Next taking $\kappa\to\infty$, we reach the following dynamics called Split Inverse Scale Space (Split ISS) in this paper.

Definition 2** (Split ISS).**

For $\kappa\to\infty$ and $\alpha\to 0$, define the differential inclusion,

[TABLE]

where $\rho(t)$ is right continuously differentiable, $\beta(t),\gamma(t)$ are right continuous, and $\rho(0)=\gamma(0)=0\in\mathbb{R}^{m}$, $\beta(0)=0\in\mathbb{R}^{p}$. Solving $\beta(t)$ in 1.8a and plugging it into 1.8b, 1.8 can be reduced to

[TABLE]

where $\Sigma$ and $A$ are given by

[TABLE]

In fact by 1.8a we have

[TABLE]

where $A=\nu X^{*}X+D^{T}D$. Substituting this for $\beta(t)$ in 1.8b and noting C.1 ($DAX^{*}=\Sigma^{1/2}\Sigma^{{\dagger}1/2}DAX^{*}$), we thus get 1.9a.

Remark 1*.*

Note that 1.9 coincides with the differential inclusion proposed in Chapter 8 of Moeller (2012) where the authors introduced it in a different way. The existence and uniquess of solutions of Split LBISS and Split ISS will be characterized precisely in Section 3.

Now consider the particular case of the standard Lasso where $D=I$ and $\Sigma(\nu)=X^{*}(I+\nu XX^{*})^{-1}X$. Hence as $\nu\to 0$, we have $\Sigma(\nu)\to X^{*}X$ and 1.9 leads to the standard Inverse Scale Space (ISS) dynamics studied in (Osher et al., 2016) by identifying $\beta=\gamma$.

Proposition 1**.**

Let $D=I$ and $\nu\to 0$, then $\gamma=\beta$ and (1.8) reduces to

[TABLE]

*with the same notations as above. *

A fundamental path consistency problem is the following.

Model Selection Consistency: Under what conditions there exists a point $\bar{\tau}$ (or $\bar{k}$) such that $\mathrm{supp}(\gamma(\bar{\tau}))=S$ (or $\mathrm{supp}(\gamma_{\bar{k}})=S$), or more specifically the so called sign-consistency holds, $\mathrm{sign}(\gamma(\bar{\tau}))=\mathrm{sign}(\gamma^{\star})$ (or $\mathrm{sign}(\gamma_{\bar{k}})=\mathrm{sign}(\gamma^{\star})$, respectively)?

Comparing the reduced Split ISS 1.9 with the ISS 1.11, one can see that $\Sigma(\nu)$ plays a similar role as $X^{*}X$. For the special case that $D=I$ and $\nu\to 0$, Osher et al. (2016) shows that under nearly the same conditions as Lasso, ISS 1.11 achieves model selection consistency but with the unbiased oracle estimator which is better than Lasso. Here an unbiased estimator means the expectation of the estimator equals to the ground truth and Lasso is well-known to be biased. In fact, under a so called Irrepresentable Condition (IRR) on $X^{*}X$, ISS 1.11 is guaranteed to evolve before the stopping time on the oracle subspace whose coordinate index is within the support set $S$ of the true parameter, i.e. no false positive. Similarly the Lasso regularization path also has no false positive under the same condition. Moreover if the signal is strong enough, the Lasso may pick up an estimator which is sign-consistent yet biased, while the ISS path with an early stopping may reach the oracle estimator which is both sign-consistent and unbiased.

For the comparison with generalized Lasso, the Irrepresentable Condition on $\Sigma(\nu)$ will replace that on $X^{*}X$, where the additional degree of freedom provided by $\nu$ enables us a chance to beat generalized Lasso.

Model selection and estimation consistency of generalized Lasso 1.2 has been studied in previous work. Sharpnack et al. (2012) considered the model selection consistency of the edge Lasso, with a special $D$ in 1.2, which has applications over graphs. Liu et al. (2013) provides an upper bound of estimation error by assuming the design matrix $X$ is a Gaussian random matrix. In particular, Vaiter et al. (2013) proposes a general condition called Identifiability Criterion (IC) for sign consistency. Lee et al. (2013) establishes a general framework for model selection consistency for penalized M-estimators, proposing an Irrepresentable Condition which is equivalent to IC from Vaiter et al. (2013) under the specific setting of 1.2. In fact both of these conditions are sufficient and necessary for structural sparse recovery by generalized Lasso 1.2 in a certain sense.

In this paper, we shall present a new family of the Irrepresentable Condition depending on $\Sigma(\nu)$, under which model selection consistency can be established for both Split ISS 1.8 and Split LBI 1.4. In particular, this condition family can be strictly weaker than IC as the parameter $\nu$ grows, which sheds light on the superb performance of Split LBI we observed in the experiment above. Therefore, the benefits of exploiting Split LBI 1.4 not only lie in its algorithmic simplicity, but also provide a possibility of theoretical improvement on model selection consistency.

Roughly speaking, the global picture of our theoretical development is illustrated in Figure 2:

Equipped with the Irrepresentable Condition on $\Sigma(\nu)$, all the dynamics (differential inclusions and the discrete iterations) evolves in a subspace of estimators whose support set lies in the that of the true parameter, whence the subspace is called the oracle subspace here; 2. 2.

Further enhanced by a restricted strongly convexity, along the paths of these dynamics the loss is rapidly decreasing at an exponential speed, firstly approaching a saddle point lying the oracle estimator then flowing away; 3. 3.

Early stopping regularization is designed here to stop the dynamics around the saddle point to pick up an estimator close to the oracle before escaping to overfitted solutions; 4. 4.

If the signal is strong enough such that the true parameters are all of sufficiently large magnitudes, such a good estimator is guaranteed to recover the sparsity pattern of the ground truth.

In the sequel, we are going to elaborate them in a precise way.

1.4 Paper Organization

This paper is a long version of a conference report (Huang et al., 2016) which states the main results about the discrete algorithm 1.4 without proofs together with part of the experiments. The full version here is organized as follows: Section 2 presents the Irrepresentable Condition together with other assumptions for Split ISS and LBI, and shows that it can be strictly weaker than IC, the necessary and sufficient condition for model selection consistency of generalized Lasso; Some basic properties of dynamic paths are presented in Section 3, including the existence and uniqueness of differential inclusion solutions, as well as the non-increasing loss along the paths; Section 4 collects the path consistency results for both differential inclusions and the discrete algorithm; A brief description of proof ideas for these results are presented in Section 5 with specific details left in appendices; Section 6 collects three more applications, including image denoising, partial order (group) estimate in sports and crowdsourced university ranking; Conclusion is given in Section 7; Appendices collect all the remaining proofs in this paper.

1.5 Notation

For matrix $Q$ with $m$ rows ($D$ for example) and $J\subseteq\{1,2,\ldots,m\}$, let $Q_{J}=Q_{J,\cdot}$ be the submatrix of $Q$ with rows indexed by $J$. However, for $Q\in\mathbb{R}^{n\times p}$ ($X$ for example) and $J\subseteq\{1,2,\ldots,p\}$, let $Q_{J}=Q_{\cdot,J}$ be the submatrix of $Q$ with columns indexed by $J$, abusing the notation.

$P_{L}$ denotes the projection matrix onto a linear subspace $L$, Let $L_{1}+L_{2}:=\{\xi_{1}+\xi_{2}:\ \xi\in L_{1},\ \xi\in L_{2}\}$ for subspaces $L_{1},L_{2}$. For a matrix $Q$, let $Q^{{\dagger}}$ denotes the Moore-Penrose pseudoinverse of $Q$, and we recall that $Q^{{\dagger}}=(Q^{T}Q)^{{\dagger}}Q^{T}$. Let $\lambda_{\max}(Q),\lambda_{\min}(Q),\lambda_{\min,+}(Q)$ denotes the largest singular value, the smallest singular value, the smallest nonzero singular value of $Q$, respectively. For symmetric matrices $P$ and $Q$, $Q\succ P$ (or $Q\succeq P$) means that $Q-P$ is positive (semi-)definite, respectively. Let $Q^{*}:=Q^{T}/n$. Sometimes we use $\langle a,b\rangle:=a^{T}b$, denoting the inner product between vectors $a,b$. Also, for tidiness in some situations, we write $(Q_{1};Q_{2}):=(Q_{1}^{T},Q_{2}^{T})^{T}$.

2 Assumptions and Comparisons

2.1 Basic Assumptions

We need some convention, definitions and assumptions. For the identifiability of $\beta^{\star}$, we can assume that $\beta^{\star}$ and its estimators of interest are restricted in

[TABLE]

since replacing $\beta^{\star}$ with “the projection of $\beta^{\star}$ onto $\mathcal{L}$” does not change the model. We also have $\beta^{\star}\in\mathcal{M}$, where $\mathcal{M}$ is the model subspace defined as

[TABLE]

Note that $\ell(\beta,\gamma)$ is quadratic, and we can define its Hessian matrix

[TABLE]

(sometimes we use the notation $H(\nu)$ stressing the dependence on $\nu$). Now we assume that there exist constants $\lambda_{D},\Lambda_{D},\Lambda_{X}>0$ satisfying

[TABLE]

Besides, we consider the following assumption.

Assumption 1** (Restricted Strong Convexity (RSC)).**

There exists a constant $\lambda>0$ such that

[TABLE]

Remark 2*.*

When $\mathcal{L}=\mathbb{R}^{p}$, i.e. there is only one $\beta^{\star}$ satisfying 1.1, 1 is the same as that proposed by Lee et al. (2013). Specifically, when $D=I$, 1 reduces to $X_{S}^{*}X_{S}\succeq\lambda I$, the usual RSC in Lasso.

Proposition 2**.**

If there exists $C>0,\ \nu>0$, such that

[TABLE]

*then 2.3 holds. Conversely, if 2.3 holds, then there exists $C>0$, such that for all $\nu>0$, 2.4 holds. *

Remark 3*.*

Traditional RSC for the partial Lasso $\min_{\beta,\gamma}(\ell(\beta,\gamma)+\lambda\|\gamma\|_{1})$ requires $\ell$ to be restricted strongly convex, i.e. strongly convex restricted on $\mathcal{N}:=\mathcal{L}\oplus\mathbb{R}^{s}\oplus\{0\}^{p-s}$ which is the sparse subspace corresponding to the support of $\gamma^{\star}$). Proposition 2 implies that, 1 is necessary for $\ell$ to be restricted strongly convex for a specific $\nu>0$ (note that $\ell$ depends on $\nu$), and also sufficient for $\ell$ to be restricted strongly convex for all $\nu>0$.

Remark 4*.*

Let us further note the rate $C/(1+\nu)$ in 2.4. When $\nu\rightarrow 0$, it approaches $C$, a constant independent with $\nu$. When $\nu\rightarrow+\infty$, the rate $C/(1+\nu)\sim\nu^{-1}$ is the best possible, since $\|H\|_{2}\lesssim\nu^{-1}$ by C.3.

Assumption 2** (Irrepresentable Condition ($\nu$) (IRR($\nu$))).**

There exists a constant $\eta\in(0,1]$ such that

[TABLE]

Remark 5*.*

2 actually concerns a family of assumptions with varing $\nu$. However, practically we only require that IRR($\nu$) holds for the specific $\nu$ used in the algorithm of Split LBI.

Remark 6*.*

2 directly generalizes the Irrepresentable Condition from standard Lasso (Zhao and Yu, 2006) and OMP/BP (Tropp, 2004), to the partial Lasso: $\min_{\beta,\gamma}(\ell\left(\beta,\gamma\right)+\lambda\|\gamma\|_{1})$. This type conditions are firstly proposed by (Tropp, 2004) for Orthogonal Matching Pursuit (OMP) and Basis Pursuit (BP) in noise free case, in the name of Exact Recovery Condition; later Cai and Wang (2011) extends it to OMP in noisy measurement; Zhao and Yu (2006) establishes it for model selection consistency of Lasso under Gaussian noise while Wainwright (2009) extends it to the sub-gaussian; Yuan and Lin (2007) and Zou (2006) also independently present this condition in other studies. Here following the standard Lasso case (Wainwright, 2009), one version of the Irrepresentable Condition should be

[TABLE]

$\rho_{(\beta,S)}^{\star}$ is the value of gradient (subgradient) of $\ell_{1}$ penalty function $\|\cdot\|_{1}$ on $(\beta^{\star};\gamma_{S}^{\star})$. Here $\rho_{\beta}^{\star}=0_{p}$, because $\beta$ is not assumed to be sparse and hence is not penalized. 2 slightly strengthens this by a supremum over $\rho$, for uniform sparse recovery independent to a particular sign pattern of $\gamma^{\star}$.

2.2 Some Equivalent Assumptions

Recall that in order to obtain path consistency results of standard LBISS and LBI in Osher et al. (2016), they propose Restricted Strong Convexity (RSC) and Irrpresentable Condition (IRR) based on their $X^{*}X$, and these assumptions are actually the same as those for Lasso. In a contrast, for Split LBISS and Split LBI, we can propose assumptions based on $\Sigma(\nu)$, i.e. $\Sigma_{S,S}(\nu)$ is positive definite, and $\|\Sigma_{S^{c},S}(\nu)\Sigma_{S,S}(\nu)^{-1}\|_{\infty}\leq 1-\eta$. These assumptions actually prove to be equivalent with 1 and 2 as follows.

Proposition 3**.**

If There exists $C>0,\ \nu_{0}>0$ such that 2.4 holds for $\nu=\nu_{0}$, then there exists $C^{\prime}>0$ such that for all $\nu>0$,

[TABLE]

*Conversely, if there exists $C^{\prime}>0,\ \nu_{0}>0$ such that 2.6 holds for $\nu=\nu_{0}$, then there exists $C>0$ such that for all $\nu>0$, 2.4 holds. *

Proposition 4**.**

Under 1, the left hand side of 2.5 in 2 becomes $\|\Sigma_{S^{c},S}(\nu)\Sigma_{S,S}(\nu)^{-1}\|_{\infty}$, and 2.5 is equivalent to

[TABLE]

Remark 7*.*

From Proposition 2 and 4, $\Sigma$ seems to be closely related to $H$, which is truly the case. In fact, $\Sigma$ is the Schur complement of $H_{\beta,\beta}$ in $H$.

2.3 Comparison Theorem on the Irrepresentable Condition

We present a comparison theorem showing that IRR($\nu$) can be weaker than IC, a necessary and sufficient for model selection consistency of generalized Lasso (Vaiter et al., 2013). Define $\mathrm{irr}(\nu)$ as the left hand side of 2.5 (or equivalently the left hand side of 2.7, due to Proposition 4), and

[TABLE]

Let $W$ be a matrix whose columns form an orthogonal basis of $\ker(D_{S^{c}})$, and define

[TABLE]

Vaiter et al. (2013) proved the sign consistency of the generalized Lasso estimator of 1.2 for specifically chosen $\lambda$, under the assumption $\mathrm{ic}_{1}<1$. As we shall see later, the same conclusion holds for our algorithm under the assumption $\mathrm{irr}(\nu)\leq 1-\eta$. Which assumption is weaker to be satisfied? The following theorem, with proof in E, answers this.

Theorem 1** (Comparisons between IRR($\nu$) and IC).**

$\mathrm{ic}_{0}\geq\mathrm{ic}_{1}$. 2. 2.

$\mathrm{irr}(0)$* exists, and $\mathrm{irr}(0)=\mathrm{ic}_{0}$.* 3. 3.

$\mathrm{irr}(\infty)$* exists, and $\mathrm{irr}(\infty)=0$ if and only if $\ker(X)\subseteq\ker(D_{S})$.*

From this comparison theorem with a design matrix $X$ of full column rank, as $\nu$ grows, $\mathrm{irr}(\nu)<\mathrm{ic}_{1}\leq\mathrm{ic}_{0}$, hence 2 is weaker than IC. Now recall the setting of Example 1 where $\ker(X)=0$ generically. In Figure 3, the (solid and dashed) horizontal red lines denote $\mathrm{ic}_{0},\mathrm{ic}_{1}$, and we see the blue curve denoting $\mathrm{irr}(\nu)$ approaches $\mathrm{ic}_{0}$ when $\nu\rightarrow 0$ and approaches [math] when $\nu\rightarrow+\infty$, which illustrates Theorem 1 (here each of $\mathrm{ic}_{0},\mathrm{ic}_{1},\mathrm{irr}(\nu)$ is the mean of $100$ values calculated under $100$ generated $X$’s). Although $\mathrm{irr}(0)=\mathrm{ic}_{0}$ is slightly larger than $\mathrm{ic}_{1}$, $\mathrm{irr}(\nu)$ can be significantly smaller than $\mathrm{ic}_{1}$ if $\nu$ is not tiny. On the right side of the vertical line, $\mathrm{irr}(\nu)$ drops below $1$, indicating that 2 is satisfied while IC fails.

Remark 8*.*

Despite that Theorem 1 suggests to adopt a large $\nu$, $\nu$ can not be arbitrarily large, elsewise $C/(1+\nu)$ in 2.4 is small and $\ell$ becomes “flat”, which will deteriorates the estimator in terms of $\ell_{2}$ error to be shown later.

3 Basic Properties of Paths

The following theorem establishes the solution existence as well as uniqueness of Split ISS and Split LBISS, in almost the same way as Osher et al. (2016). The proof is given in C.

Theorem 2** (Existence and uniqueness of solutions).**

As for Split ISS 1.8, assume that $\rho(t)$ is right continuously differentiable and $\beta(t),\gamma(t)$ is right continuous. Then a solution exists for $t\geq 0$, with piecewise linear $\rho(t)$ and piecewise constant $\beta(t),\gamma(t)$. Besides, $\rho(t)$ is unique. If additionally $\Sigma_{S(t),S(t)}\succ 0$ for $0\leq t\leq\tau$, where $\Sigma$ is defined in 1.10 and $S(t):=\mathrm{supp}(\gamma(t))$, then $\beta(t),\gamma(t)$ are unique for $0\leq t\leq\tau$. 2. 2.

As for Split LBISS 1.7, assume that $\rho(t),\beta(t)$ are right continuously differentiable. Then there is a unique solution for $t\geq 0$.

The following theorem states that along the solution path of either differential inclusions or iterative algorithms, the loss function is always non-increasing. Its proof is provided in C.

Theorem 3** (Non-increasing loss along the paths).**

Consider the loss function $\ell$ defined in 1.3.

For a solution $(\rho(t),\beta(t),\gamma(t))$ of Split ISS 1.8, $\ell(\beta(t),\gamma(t))$ is non-increasing in $t$. 2. 2.

For a solution $(\rho(t),\beta(t),\gamma(t))$ of Split LBISS 1.7, $\ell(\beta(t),\gamma(t))$ is non-increasing in $t$. 3. 3.

For a solution $(\rho_{k},\beta_{k},\gamma_{k})$ of Split LBI 1.6, $\ell(\beta_{k},\gamma_{k})$ is non-increasing in $k$, if

[TABLE]

Moreover, $\left\|H\right\|_{2}\leq 2\left(1+\nu\Lambda_{X}^{2}+\Lambda_{D}^{2}\right)/\nu$ holds, so 3.1 holds if

[TABLE]

4 Path Consistency of Split LBISS and Split LBI

4.1 Consistency of Split LBISS

The following theorem, with proof in G, says that under 1 and 2, Split LBISS will automatically evolve in the “oracle” subspace (unknown to us) restricted within the support set of $(\beta^{\star},\gamma^{\star})$ before leaving it, and if the signal parameters is strong enough, sign consistency will be reached. Moreover, $\ell_{2}$ error bounds on $\gamma(t)$ and $\beta(t)$ are given.

Theorem 4** (Consistency of Split LBISS).**

Under 1 and 2, define $\lambda_{H}=C/(1+\nu)$ (from 2.4) and suppose that $\kappa$ is large so that

[TABLE]

Let

[TABLE]

Then with probability not less than $1-6/m-3\exp(-4n/5)$, we have all the following properties.

No-false-positive*: The solution has no false-positive, i.e. $\mathrm{supp}(\gamma(t))\subseteq S$, for $0\leq t\leq\overline{\tau}$.* 2. 2.

Sign consistency of $\gamma(t)$**: Once the signal is strong enough such that

[TABLE]

then $\gamma(t)$ has sign consistency at $\bar{\tau}$, i.e. $\mathrm{sign}(\gamma(\bar{\tau}))=\mathrm{sign}(D\beta^{\star})$. 3. 3.

$\ell_{2}$ consistency of $\gamma(t)$**:

[TABLE] 4. 4.

$\ell_{2}$ “consistency” of $\beta(t)$**:

[TABLE]

where

[TABLE]

which is very small in most cases.

Despite that the sign consistency of $\gamma(t)$ can be established here, usually one can not expect $D\beta(t)$ recovers the sparsity pattern of $\gamma^{\star}$ due to the variable splitting. As shown in the last term of the $\ell_{2}$ error bound of $\beta(t)$, increasing $\nu$ will sacrifice its accuracy, as to achieve the minimax optimal $\ell_{2}$ error rate one needs $\nu=O(\sqrt{(s\log m)/n})$. However, one can remedy this by projecting $\beta(t)$ on to a subspace using the support set of $\gamma(t)$, and obtain a good estimator $\tilde{\beta}(t)$ with both sign consistency and $\ell_{2}$ consistency at the minimax optimal rates. This leads to the following theorem.

Theorem 5** (Consistency of revised Split LBISS).**

Under 1 and 2, define $\lambda_{H}=C/(1+\nu)$ (from 2.4) and suppose that $\kappa$ satisfies 4.1. Define $\bar{\tau}$ the same as in Theorem 4, and define

[TABLE]

If $S(t)^{c}=\varnothing$, define $P_{S(t)}=I$. Then we have the following properties.

Sign consistency of $\tilde{\beta}(t)$**: Once 4.3 holds, then with probability not less than $1-8/m-3\exp(-4n/5)$, there holds $\mathrm{sign}(D\tilde{\beta}(\bar{\tau}))=\mathrm{sign}(D\beta^{\star})$. 2. 2.

$\ell_{2}$ consistency of $\tilde{\beta}(t)$**: With probability not less than $1-8/m-2r^{\prime}/m^{2}-3\exp(-4n/5)$, we have

[TABLE]

where $r^{\prime}$ is defined in 4.4. If additionally $S(\bar{\tau})\supseteq S$, then the last term on the right hand side drops.

Remark 9*.*

In most cases $r^{\prime}$ is very small, so the dominant $\ell_{2}$ error rate is $O(\sqrt{(s\log m)/n})$ (as long as $\nu$ is upper bounded by constant), which is minimax optimal (Lee et al., 2013; Liu et al., 2013).

4.2 Consistency of Split LBI

Based on theorems on consistency of Split LBISS, one can naturally derive similar results for Split LBI with large $\kappa$ and small $\alpha$.

Theorem 6** (Consistency of Split LBI).**

Under 1 and 2, define $\lambda_{H}=C/(1+\nu)$ (from 2.4). Suppose that $\kappa$ is large and $\alpha$ is small, so that

[TABLE]

$\kappa$* satisfies 4.1 with $\lambda_{H}$ replaced by $\lambda_{H}^{\prime}:=\lambda_{H}(1-\kappa\alpha\|H\|_{2}/2)>0$, and*

[TABLE]

Let $\overline{k}:=\lfloor\bar{\tau}/\alpha\rfloor$. Then with probability not less than $1-6/m-3\exp(-4n/5)$, we have all the following properties.

No-false-positive*: The solution has no false-positive, i.e. $\mathrm{supp}(\gamma_{k})\subseteq S$, for $0\leq k\alpha\leq\overline{\tau}$.* 2. 2.

Sign consistency of $\gamma_{k}$**: Once the signal is strong enough such that

[TABLE]

then $\gamma_{k}$ has sign consistency at $\bar{k}$, i.e. $\mathrm{sign}(\gamma_{\bar{k}})=\mathrm{sign}(D\beta^{\star})$. 3. 3.

$\ell_{2}$ consistency of $\gamma_{k}$**:

[TABLE] 4. 4.

$\ell_{2}$ “consistency” of $\beta_{k}$**:

[TABLE]

where $r^{\prime}$ is defined in 4.4.

Similarly after the projection one can get $\tilde{\beta}_{k}$ such that $D\tilde{\beta}_{k}$ is sparse and the corresponding $\ell_{2}$ error bound is improved.

Theorem 7** (Consistency of revised Split LBI).**

Under 1 and 2, define $\lambda_{H}=C/(1+\nu)$ (from 2.4). Suppose that $\kappa,\alpha$ satisfy the same conditions as in Theorem 6; $\lambda_{H}^{\prime},\bar{\tau}$ is defined the same as in Theorem 6. Define

[TABLE]

If $S_{k}^{c}=\varnothing$, define $P_{S_{k}}=I$. Then we have the following properties.

Sign consistency of $\tilde{\beta}_{k}$**: Once 4.6 holds, then with probability not less than $1-8/m-3\exp(-4n/5)$, there holds $\mathrm{sign}(D\tilde{\beta}_{\bar{k}})=\mathrm{sign}(D\beta^{\star})$. 2. 2.

$\ell_{2}$ consistency of $\tilde{\beta}_{k}$**: With probability not less than $1-8/m-2r^{\prime}/m^{2}-3\exp(-4n/5)$, we have

[TABLE]

where $r^{\prime}$ is defined in 4.4. If additionally $S_{\bar{k}}\supseteq S$, then the last term on the right hand side drops.

5 Proof Ideas for SLBISS Path Consistency Theorems

Sketchy proof of Theorem 4.

The Split LBISS dynamics always start within the oracle subspace ($\gamma_{S^{c}}(t)=0$), and by Lemma 7 we prove that under the Irrepresentable Condition the exit time of the oracle subspace is no earlier than some $\bar{\tau}\lesssim\sqrt{n/\log m}$ (i.e. the no-false-positive condition holds before $\bar{\tau}$), with high probability.

Before $\bar{\tau}$, the dynamics follow the identical path of the following oracle dynamics of the Split LBISS restricted in the oracle subspace

[TABLE]

where $\rho_{S}^{\prime}(0)=\gamma_{S}^{\prime}(0)=0\in\mathbb{R}^{s},\ \beta^{\prime}(0)=0\in\mathbb{R}^{p}$. Theorem 3 shows that the loss is always dropping along the paths. Hence to monitor the distance of an estimator to the oracle estimator

[TABLE]

which222The property of the right hand side of 5.2 is based on $\ell(P_{\mathcal{L}}\beta^{o},\gamma^{o})=\ell(\beta^{o},\gamma^{o})$. is an optimal estimate of the true parameter $(\beta^{\star},\gamma^{\star})$ (with error bounds in Lemma 8), we define a potential function

[TABLE]

where

[TABLE]

and the Bregman distance

[TABLE]

Equipped with this potential function, the original differential inclusion is reduced to the following differential inequality, called as generalized Bihari’s inequality (Lemma 1) whose proof will be given in F.

Lemma 1** (Generalized Bihari’s inequality).**

Under 1, for all $t\geq 0$ we have

[TABLE]

where $\gamma_{\min}^{o}:=\min(|\gamma_{j}^{o}|:\ \gamma_{j}^{o}\neq 0)$ and

[TABLE]

Such an inequality, together with the Restricted Strong Convexity condition (RSC), leads to an exponential decrease of the potential above enforcing the convergence to the oracle estimator. Then we can show that as long as the signal is strong enough with all the magnitudes of entries of $\gamma^{\star}$ being large enough ($\gtrsim(\log s)\sqrt{(\log m)/n}$), the dynamics stopped at $\bar{\tau}$, exactly selects all nonzero entries of $\gamma^{o}$ (F.8 in Lemma 6), hence also of $\gamma^{\star}$ with high probability, achieving the sign consistency.

Even without the strong signal condition, with RSC we can also show that the dynamics, at $\bar{\tau}$, returns a good estimator of $\gamma^{o}$ (F.9 in Lemma 6), hence also of $\gamma^{\star}$, having an $\ell_{2}$ error $\simeq\sqrt{(s\log m)/n}$ (minimax optimal rate) with high probability. Combining the $\ell_{2}$ bounds of $\beta^{\prime}(t)-\beta^{o}$ (from F.9 in Lemma 6) and $\beta^{o}-\beta^{\star}$ (Lemma 8), we obtain the result concerning the $\ell_{2}$ bound of $\beta^{\prime}(t)-\beta^{\star}$, at $\bar{\tau}$, similarly with the minimax optimal rate.

A detailed proof of Theorem 4 can be found in G.

Remark 10*.*

It is an interesting open problem how to relax the Irrepresentable Condition to achieve a minimax optimal estimator at weaker conditions such as (Bickel et al., 2009).

Proof sketch of Theorem 5.

By Theorem 4, the exit time of the oracle subspace is no earlier than some $\bar{\tau}\lesssim\sqrt{n/\log m}$, i.e. the no-false-positive condition holds before $\bar{\tau}$, or say $S(t)\subseteq S$ for $t\leq\bar{\tau}$, with high probability. The definition of $\tilde{\beta}(t)$ enforces

[TABLE]

Using the error bounds of $\beta^{\prime}(t)-\beta^{o}$ (from F.8 in Lemma 6) and $\beta^{o}-\beta^{\star}$ (Lemma 8), we obtain

[TABLE]

as long as the magnitudes of entries of $\gamma^{\star}$ are all large enough, achieving the sign consistency. Also we can obtain the $\ell_{2}$ bound of $\tilde{\beta}(t)-\beta^{\star}$.

A detailed proof of Theorem 5 can be found in G.

6 Experiments

In this section, we show three additional applications using the algorithm proposed in this paper. The first application is about traditional image denoising using TV-regularization or fused Lasso. The remaining twos are new applications in partial order ranking: the second one is the basketball team ranking in partial order and the third one is the grouping of world universities in crowdsourced ranking. For reproducible research, Matlab source codes are released at the following website:

https://github.com/yuany-pku/split-lbi.

6.1 Parameter Setting

Parameter $\kappa$ should be large enough according to 4.1. Moreover, step size $\alpha$ should be small enough to ensure the stability of Split LBI. When $\nu,\kappa$ are determined, $\alpha$ can actually be determined by $\alpha=\nu/(\kappa(1+\nu\Lambda_{X}^{2}+\Lambda_{D}^{2}))$ (see 3.2).

6.2 Application: Image Denoising

Consider the image denoising problem in Tibshirani and Taylor (2011). The original image is resized to $50\times 50$, and reset with only four colors, as in the top left image in Figure 4. Some noise is added by randomly changing some pixels to be white, as in the bottom left. Let $G=(V,E)$ is the 4-nearest-neighbor grid graph on pixels, then $\beta=(\beta_{R};\beta_{G};\beta_{B})\in\mathbb{R}^{3|V|}$ since there are 3 color channels (RGB channels). $X=I_{3|V|}$ and $D=\mathrm{diag}(D_{G},D_{G},D_{G})$, where $D_{G}\delta\in\mathbb{R}^{|E|\times|V|}$ is the gradient operator on graph $G$ defined by $(D_{G}x)(e_{ij})=x_{i}-x_{j},\ e_{ij}\in E$. Set $\nu=180,\ \kappa=100$. The regularization path of Split LBI is shown in Figure 4, where as $t$ evolves, images on the path gradually select visually salient features before picking up the random noise.

Now compare the AUC (Area Under the Curve) of genlasso and Split LBI algorithm with different $\nu$. For simplicity we show the AUC corresponding to the red color channel. Here $\nu\in\{1,20,40,60,\ldots,300\}$. As shown in the right panel of Figure 4, with the increase of $\nu$, Split LBI beats genlasso with higher AUC values.

6.3 Application: Partial Order Ranking for Basketball Teams

Here we consider a new application on the ranking of $p=12$ FIBA basketball teams into partial orders. The teams are listed in Figure 5. We collected $n=134$ pairwise comparison game results mainly from various important championship such as Olympic Games, FIBA World Championship and FIBA Basketball Championship in 5 continents from 2006–2014 (8 years is not too long for teams to keep relatively stable levels while not too short to have enough samples). For each sample indexed by $k$ and corresponding team pair $(i_{k},j_{k})$, $y_{k}=s_{i_{k}}-s_{j_{k}}$ is the score difference between team $i_{k}$ and $j_{k}$. We assume a model $y_{k}=\beta^{\star}_{i_{k}}-\beta^{\star}_{j_{k}}+\epsilon_{k}$ where $\beta^{\star}\in\mathbb{R}^{p}$ measures the strength of these teams. So the design matrix $X\in\mathbb{R}^{n\times p}$ is defined by its $k$-th row: $x_{k,i_{k}}=1,\ x_{k,j_{k}}=-1,\ x_{k,l}=0\ (l\neq i_{k},j_{k})$. In sports, teams with similar strength generally meet more often than those in different levels. Thus we hope to find a coarse grained partial order ranking by adding a structural sparsity on $D\beta^{\star}$ where $D=cX$ ($c$ scales the smallest nonzero singular value of $D$ to be 1).

The top left panel of Figure 5 shows $\{\beta_{\lambda}\}$ by genlasso and $\tilde{\beta}_{k}$ by Split LBI with $\nu=1$ and $\kappa=100$. Both paths give the same partial order at early stages, though the Split LBI path looks qualitatively better. For example, the top right panel shows the same partial order after the change point $t_{5}$. It is interesting to compare it against the FIBA ranking in September, 2014, shown in the bottom. Note that the average basketball level in Europe is higher than that of in Asia and Africa, hence China can get more FIBA points than Germany based on the dominant position in Asia, so is Angola in Africa. But their true levels might be lower than Germany, as indicated in our results. Moreover, America (FIBA points $1040.0$) itself forms a group, agreeing with the common sense that it is much better than any other country. Spain, having much higher FIBA ranking points ($705.0$) than the 3rd team Argentina ($455.0$), also forms a group alone. It is the only team that can challenge America in recent years, and it enters both finals against America in 2008 and 2012.

6.4 Application: Grouping in Crowdsourced Ranking of World Universities

Crowdsourcing technique has been recently used to rank universities by Internet voters, e.g. CrowdRank. In the following a crowdsourcing experiment has been conducted for ranking $p=261$ universities in the world on the platform http://www.allourideas.org/worldcollege. The majority of the participants are undergraduates or alumni from Peking University, mostly majoring in applied mathematics and statistics while some with engineering background. Voters are widely distributed around the world, with one fifth of all from Beijing, see Figure 6. Every voter is presented with a randomly chosen pair of universities, and asked with the question “which university would you rather attend?”. Then the voter is allowed to choose either of the two universities, or simply “I can’t decide”. Our collection consists of about eight thousand votes. To make our result more robust, we remove some indecisive votes or outliers using the technique from Xu et al. (2014) and are left with $n=6,125$ paired comparison samples in the cleaned dataset for the study in this paper. For each sample indexed by $k$ and corresponding university pair $(i_{k},j_{k})$, if the voter considers $i_{k}$ to be better than $j_{k}$, then $y_{k}=1$, otherwise $y_{k}=-1$. We assume a model $y_{k}=\beta^{\star}_{i_{k}}-\beta^{\star}_{j_{k}}+\epsilon_{k}$ where $\beta^{\star}\in\mathbb{R}^{p}$ measures the strength of these universities. So the design matrix $X\in\mathbb{R}^{n\times p}$ is defined by its $k$-th row: $x_{k,i_{k}}=1,\ x_{k,j_{k}}=-1,\ x_{k,l}=0\ (l\neq i_{k},j_{k})$. $D$ is denoted as the total variation matrix with complete graph, i.e. $\|D\beta\|_{1}=\Sigma_{i<j}|\beta_{i}-\beta_{j}|_{1}$ for any $\beta\in R^{p}$. Split LBI is then implemented to obtain $\tilde{\beta}_{k}$.

Similar to the the previous application on basketball team ranking, for each $k$, entries of $\tilde{\beta}_{k}$ with same values form a group. For each $k$, consider the directed graph $G_{t}=(V,E_{t})\ (t=k\alpha)$ with $V=\{1,\ldots,p\}$ and $E_{t}$ consisting of directed edges $(i,j)$’s with $\tilde{\beta}_{k,i}\neq\tilde{\beta}_{k,j}$ ($i\to j$ if $\tilde{\beta}_{k,i}<\tilde{\beta}_{k,j}$). We pick up $5$ universities for a simple illustration. See Figure 7 for the path and corresponding graphs for $t=t_{1},\ldots,t_{4}$. No edge is selected at $t=0$. At $t=t_{1}$, Beijing Forestry University is left behind. At $t=t_{2}$, we see that Harvard University and The University of Cambridge form the 1st group; Peking University and The University of Tokyo form the 2nd group; Beijing Forestry University becomes the last group. Continuing at $t=t_{3},t_{4}$, further refinements within the 1st and 2rd groups are made. Note that at $t=t_{4}$, Peking University is more preferred to The University of Tokyo, yet below Harvard and Cambridge, reflecting the preference of voters from Peking University.

Now back to the whole set of $p=261$ universities, we pick up a particular time at which the universities are separated into $10$ groups. Some reasonable results can be observed. See Table 2 for the 1st group consisting of $17$ top universities. Most of them are first tier universities in USA, together with two top universities in UK (University of Cambridge, University of Oxford). California is clearly a favorite place for these voters, having five institutes included in the first group.

The 2nd group universities are listed in See Table 3. It includes top universities in Asia (Peking University, Tsinghua University, University of Tokyo, University of Hong Kong, and Hong Kong University of Science and Technology), Europe (Swiss Federal Institute of Technology/ETH, Imperial College London, University College London, and London School of Economics and Political Science), and North America. A surprising result is that MIT is listed in this second group, while most of the authoritative ranking systems clearly place it in the first tier. This phenomenon is probably due to the sampling bias in our crowdsourcing experiment: a large portion of the voters are of statistics major and MIT does not have a statistics department or program. Hence such voters will not choose MIT when considering graduate programs.

Information about other groups can be found on website: https://github.com/yuany-pku/split-lbi/tree/master/examples/university.

7 Conclusion

In this paper, we introduce a novel iterative regularization path with structural sparsity such that parameters are sparse under certain linear transform. Variable splitting is exploited to lift the parameters into a high dimensional space with separate parameters for data fitting and sparse model selection. A statistical benefit of such a splitting lies in its improved model selection consistency under weaker conditions than the traditional generalized Lasso, shown in both theory and experiments. For the statistical analysis of such an algorithm, several limit dynamics as differential inclusions are introduced which sheds light on the consistency properties of the regularization paths. Finally some applications are given with real world data, including image denoising, partial order ranking of basket ball teams, and grouping of world universities by crowdsourced ranking. These results show that the benefit of the proposed algorithm lies in both its simplicity in computing the regularization path iteratively and its solid theoretical guarantee on path consistency. Hence it can be regarded as a generalization of $L_{2}$Boost in machine learning or Landweber iteration in inverse problems with structural sparsity control.

Appendix A Further Notations throughout the Appendix

Apart from Section 1.5 and 2.1, we need more notations throughout the appendix. Let the compact singular value decomposition (compact SVD) of $D$ be

[TABLE]

and $(V,\tilde{V})$ be an orthogonal square matrix. Let the compact SVD of $X\tilde{V}/\sqrt{n}$ be

[TABLE]

and let $(V_{1},\tilde{V}_{1})$ be an orthogonal square matrix. $r^{\prime}$ in A.2 is the rank of $X\tilde{V}$, which meets the definition 4.4.

We have $\lambda_{D}I\preceq\Lambda\preceq\Lambda_{D}I$. If $\ker(D)\subseteq\ker(X)$ (for example, $D$ has full column rank), then $X\tilde{V}=0$, and $\tilde{V},U_{1},\Lambda_{1},V_{1},\tilde{V}_{1}$ all drop.

Generally, $r^{\prime}\leq p-r$. If 1 holds, noting for any $\xi\in\mathbb{R}^{p-r}$, $\tilde{V}\xi\in\ker(D)\subseteq\mathcal{M}$, and

[TABLE]

we have $V_{1}\Lambda_{1}^{2}V_{1}^{T}=\tilde{V}^{T}X^{*}X\tilde{V}\succeq\lambda I$. Since $V_{1}$ is a tall matrix, we further know it is square (elsewise $V_{1}\Lambda_{1}^{2}V_{1}^{T}$ is not invertible), i.e. $r^{\prime}=p-r$. Besides, we have $\lambda_{1}:=\lambda_{\min}(\Lambda_{1})\geq\sqrt{\lambda}$ (when $\Lambda_{1}$ drops, $\lambda_{1}:=+\infty$).

From now, we also write $\lambda_{H}=C/(1+\nu),\ \lambda_{\Sigma}=C^{\prime}/(1+\nu)$ according to Proposition 2 and 3.

Appendix B Some Useful Technical Lemmas

Lemma 2** (Concentration inequalities).**

Suppose that $\epsilon\in\mathbb{R}^{n}$ has independent identically distributed components, each of which has a sub-Gaussian distribution with parameter $\sigma^{2}$, i.e. $\mathbb{E}[\exp(t\epsilon_{i})]\leq\exp(\sigma^{2}t^{2}/2)$, then

[TABLE]

Moreover, by B.1 we have that for $B\in\mathbb{R}^{q\times n}$, with probability not less than $1-2q/m^{2}$,

[TABLE]

By B.2 we have that with probability not less than $1-\exp(-4n/5)$,

[TABLE]

Proof.

As for B.1, let $B=(B_{i,j})_{q\times n}$ and $1\leq i\leq q$, it is well-known that

[TABLE]

is also sub-Gaussian, with parameter $b_{i}^{2}=(B_{i,1}^{2}+\cdots+B_{i,n}^{2})\sigma^{2}$. Thus

[TABLE]

As for B.2, note that for $0\leq\zeta<1/2$,

[TABLE]

Take $\zeta=z/(2(1+z))\in[0,1/2)$, and B.2 follows.

Lemma 3** (Transformations and bounds for quadratic forms).**

If

[TABLE]

then

[TABLE]

Moreover, for $0\leq\zeta\leq 1$, the following two statements are equivalent:

[TABLE]

And if B.6 and B.7 hold, then by B.5 we easily obtain

[TABLE]

Proof.

Theorem 1.19 in Zhang (2006) tells that $PP^{{\dagger}}Q=Q$, so it is easy to verify

[TABLE]

Thus

[TABLE]

Similarly we can obtain another inequality.

If B.6 holds, then

[TABLE]

By Theorem 1.19 in Zhang (2006) we have $QR^{{\dagger}1/2}R^{1/2}=Q$, thus $Q^{T}P^{{\dagger}}Q\preceq(1-\zeta)R$, i.e. B.7 holds. Similarly B.7 implies B.6.

Lemma 4** (Representation of $\mathcal{L}$).**

Adopt notations from A.1 and A.2. $\beta\in\mathcal{L}$ if and only if

[TABLE]

Proof.

Note that

[TABLE]

Right multiplying $\beta$ on both side leads to

[TABLE]

It suffices to show $\ker\left(\tilde{V}_{1}^{T}\tilde{V}^{T}\right)=\mathcal{L}$, which is equivalent to

[TABLE]

For any $\beta\in\mathcal{L}^{\prime}$, we have $X\beta=0,\ D\beta=0$ since $X\tilde{V}\tilde{V}_{1}=0,\ D\tilde{V}=0$, so $\beta\in\mathcal{L}^{\perp}$. Conversely, if $\beta\in\mathcal{L}^{\perp}$, left multiplying $D$ on both sides of B.9 leads to $\delta=0$. Then left multiplying $X$ on both sides of B.9 further leads to $\xi=0$. Now B.9 tells that $\beta\in\mathcal{L}^{\prime}$. So $\mathcal{L}^{\prime}=\mathcal{L}^{\perp}$.

Lemma 5**.**

Adopt the notation from A.1 and A.2. Define

[TABLE]

We have

[TABLE]

Consequently,

[TABLE]

Proof.

Note that

[TABLE]

We can directly verify that $(QMQ^{T})^{{\dagger}}=(Q^{T})^{-1}M^{{\dagger}}Q^{-1}$, thus

[TABLE]

which comes to be the right hand side of B.10. Now it is easy to verify B.11.

Appendix C Proof on Basic Path Properties of Split ISS, Split LBISS and Split LBI

Proof of Theorem 2.

For Split ISS, by 1.8a and the fact that $\beta(t)\in\mathcal{L}=\mathrm{Im}(X^{T})+\mathrm{Im}(D^{T})=\mathrm{Im}(A)=\mathrm{Im}(A^{{\dagger}})$, we can solve $\beta(t)=A^{{\dagger}}(\nu X^{*}y+D^{T}\gamma(t))$ which is determined by $\gamma(t)$. Plugging it into 1.8b we have

[TABLE]

Taking $M=I_{p+m}-(\sqrt{\nu/n}X^{T},D^{T})^{{\dagger}}(\sqrt{\nu/n}X^{T},D^{T})$ in Theorem 1.19 in Zhang (2006) leads to

[TABLE]

The inclusion becomes

[TABLE]

which is a standard ISS (on $\gamma(t)$) and has been sufficiently discussed in Osher et al. (2016) (let $X,y$ in that paper take $\sqrt{n}\Sigma^{1/2}$ and $\sqrt{n}\Sigma^{{\dagger}1/2}DA^{{\dagger}}X^{*}y$ in this paper). Specifially, there exists a solution with piecewise linear $\rho(t)$ and piecewise constant $\beta(t),\gamma(t)$. Besides, $\rho(t)$ is unique. If additionally, when $\Sigma_{S(t),S(t)}\succ 0$, we have that $\Sigma_{\cdot,S(t)}$ has full column rank, and $\gamma(t)$ (hence $\beta(t)$) is unique.

For Split LBISS, letting $z(t)=\rho(t)+\gamma(t)/\kappa$ and noting 1.5, the Split LBISS 1.7 is equivalent to

[TABLE]

The Picard-Lindelöf Theorem implies that this ODE has a unique solution $(\beta(t),z(t))$, so there exists a unique solution to the Split LBISS 1.7.

Proof of Theorem 3.

For Split ISS, one can easily imitates the technique in the proof of Theorem 2.1 in Osher et al. (2016) to show that $(\beta(t),\gamma(t))$ is the solution of the following optimization problem.

[TABLE]

for any $t>0$, due to the continuity of $\rho(\cdot)$, there is a small neighborhood of $t$, on which every $\tau$ satisfies

[TABLE]

That is to say, $(\beta(\tau),\gamma(\tau))$ satisfies the constraints in C.2, so the value of $\ell(\beta(\tau),\gamma(\tau))$ is not less than $\ell(\beta(t),\gamma(t))$ (the solution of C.2). This implies that any $t\geq 0$ is a local minimal point of a right continuous function $\ell(\beta(\cdot),\gamma(\cdot))$. Then by standard techniques in mathematical analysis, we have that $\ell(\beta(t),\gamma(t))$ is non-increasing.

For Split LBISS, by 1.7c, we have $\dot{\gamma}_{j}(t)\cdot\dot{\rho}_{j}(t)\equiv 0$ for each $j$, so $\ell$ is non-increasing since

[TABLE]

For Split LBI, noting $(\rho_{k+1}-\rho_{k})(\gamma_{k+1}-\gamma_{k})=\|\rho_{k+1}\|_{1}-\langle\rho_{k+1},\gamma_{k}\rangle+\|\gamma_{k+1}\|_{1}-\langle\rho_{k},\gamma_{k+1}\rangle\geq 0$, we have

[TABLE]

By $\kappa\alpha\|H\|_{2}<2$, we have

[TABLE]

Moreover, it is easy to verify that

[TABLE]

Appendix D Proof on Equivalence of Assumptions

Proof of Proposition 2.

If there exists $C>0,\ \nu>0$ such that 2.4 holds, then for $\beta\in\mathcal{L}\cap\mathcal{M}$, taking $\gamma_{S}=D_{S}\beta$ and noting $D_{S^{c}}\beta=0$, the left hand side of 2.4 is

[TABLE]

which should be not less than $(C/(1+\nu))\|\beta\|_{2}^{2}$. Thus 1 holds for $\lambda=C/(1+\nu)>0$.

If 1 holds for some $\lambda>0$, for any $\beta\in\mathcal{L}$, let $\beta=\beta^{\prime}+\beta^{\prime\prime}$ where $\beta^{\prime}\in\mathcal{L}\cap\mathcal{M}$ and $\beta^{\prime\prime}\in\mathcal{L}\cap\mathcal{M}^{\perp}$. Since $D_{S^{c}}\beta^{\prime}=0,\ \beta^{\prime\prime}\in\mathcal{M}^{\perp}=\mathrm{Im}(D_{S^{c}}^{T})$, we have

[TABLE]

For constant $\nu_{0}=2\lambda_{D}^{2}/(\lambda+2\Lambda_{X}^{2})>0$ we have

[TABLE]

The left hand side of 2.4, denoted by $L$ or $L(\nu)$, satisfies

[TABLE]

Furthermore, by the inequality above and Cauchy’s inequality,

[TABLE]

Consequently,

[TABLE]

where

[TABLE]

is a constant. Thus 2.4 holds for all $\nu>0$.

Proof of Proposition 3.

Let $L=L(\nu)$ denotes the left hand side of 2.4. Suppose there exists $C>0,\ \nu_{0}>0$ such that 2.4 holds for $\nu=\nu_{0}$. Since

[TABLE]

we can find $C_{1}>0$ such that 2.4 holds for $C=C_{1}$ and all $\nu>0$. Now

[TABLE]

So

[TABLE]

$L\geq(C_{1}/(1+\nu))\|(\beta;\gamma_{S})\|_{2}^{2}$ implies $\gamma_{S}^{T}\Sigma_{S,S}\gamma_{S}\geq(C_{1}/(1+\nu))\|\gamma_{S}\|_{2}^{2}$ (letting $\beta=A^{{\dagger}}D_{S}^{T}\gamma_{S}\in\mathcal{L}$). So 2.6 holds for $C^{\prime}=C_{1}$ and all $\nu>0$.

Suppose there exists $C^{\prime}>0,\ \nu_{0}>0$ such that 2.6 holds for $\nu=\nu_{0}$. For any $\beta\in\mathcal{L}$, represent $\beta=V\delta+\tilde{V}V_{1}\xi$ by Lemma 4, then

[TABLE]

For $P=\Lambda^{2}+\nu_{0}V^{T}X^{*}XV,\ Q=\nu_{0}V^{T}X^{T}U_{1}\Lambda_{1}/\sqrt{n},\ R=\nu_{0}\Lambda_{1}^{2}$,

[TABLE]

By Lemma 3 we have

[TABLE]

Note that

[TABLE]

By Lemma 3 we have

[TABLE]

Thus

[TABLE]

Where $C_{1}^{\prime}>0$ is a constant. Besides,

[TABLE]

Thus we can find $C_{2}^{\prime}>0$ such that $L(\nu_{0})\geq C_{2}^{\prime}\left\|(\beta;\gamma_{S})\right\|_{2}^{2}$. Combining with D.2, we can find $C>0$ such that 2.4 holds for all $\nu>0$.

Proof of Proposition 4.

Under 1, by Proposition 2 and 3 we have $\Sigma_{S,S}\succ 0$. By D.3, we know

[TABLE]

Then by Theorem 1.21 in Zhang (2006), we have that

[TABLE]

By $H_{S^{c},(\beta,S)}=(-D_{S^{c}}/\nu,0)$ and $-D_{S^{c}}A^{{\dagger}}D_{S}/\nu=\Sigma_{S^{c},S}$, we have

[TABLE]

The rest is easy.

Appendix E Proof of the Comparison Theorem

Proof of Theorem 1.

By definition, we have $\mathrm{ic}_{0}\geq\|\Omega^{S}\mathrm{sign}(D_{S}\beta^{\star})\|_{\infty}\geq\mathrm{ic}_{1}$. Now we prove $\mathrm{irr}(0)$ exists and $\mathrm{irr}(0)=\mathrm{ic}_{0}$. Let $M:=\Lambda^{-1}V^{T}X^{*}(I-U_{1}U_{1}^{T})XV\Lambda^{-1}$. When $\nu$ is small, by B.11,

[TABLE]

Let $F:=I-U_{S}U_{S}^{T}$ and $F=U^{\prime}\Lambda^{\prime}U^{\prime T}$ be the “compact” eigendecomposition of $F$ ($\Lambda^{\prime}\succ 0$). Let $G:=U_{S}MU_{S}^{T}$. Suppose $(U^{\prime},\tilde{U}^{\prime})$ is an orthogonal square matrix, and

[TABLE]

By $F+\nu G\succ 0$, we have $K_{3}\succ 0$. Now

[TABLE]

Define $Q_{\nu}=K_{3}-\nu K_{2}^{T}(\Lambda^{\prime}+\nu K_{1})^{-1}K_{2},\ R_{\nu}=K_{2}^{T}(\Lambda^{\prime}+\nu K_{1})^{-1}$, and we can calculate

[TABLE]

Note that $Q_{\nu}\rightarrow K_{3},R_{\nu}\rightarrow K_{2}^{T}\Lambda^{\prime-1}$, and note that

[TABLE]

Combining it with the representation of $(F+\nu G)^{-1}$,

[TABLE]

Besides,

[TABLE]

So when $\nu\rightarrow 0$,

[TABLE]

The infinity norm of the right hand side is $\mathrm{irr}(0)$. On the other hand,

[TABLE]

In order to prove $\mathrm{irr}(0)=\mathrm{ic}_{0}$, it suffices to show

[TABLE]

The first term of the right hand side is

[TABLE]

while by the fact that

[TABLE]

the second term becomes

[TABLE]

So it suffices to show

[TABLE]

which is equivalent to

[TABLE]

First we prove

[TABLE]

In fact, by E.1 we have $\mathrm{Im}(U_{S}^{T}\tilde{U}^{\prime})\subseteq\ker(U_{S^{c}})$. For any $\zeta\in\ker(U_{S^{c}})$, we have $(I-U_{S}^{T}U_{S})\zeta=U_{S^{c}}^{T}U_{S^{c}}\zeta=0$. Let

[TABLE]

then

[TABLE]

which implies $\zeta_{2}\in\mathrm{Im}(U_{S}^{T})$. But $\zeta_{2}\in\ker(U_{S})$, then $\zeta_{2}=0$, and $0=(I-U_{S}^{T}U_{S})U_{S}^{T}\zeta_{1}=U_{S}^{T}(I-U_{S}U_{S}^{T})\zeta_{1}=U_{S}^{T}U^{\prime}\Lambda^{\prime}U^{\prime T}\zeta_{1}$. Assume that $\zeta_{1}=U^{\prime}\zeta_{3}+\tilde{U}^{\prime}\tilde{\zeta}_{3}$, then $U_{S}^{T}U^{\prime}\Lambda^{\prime}\zeta_{3}=0$. Thus

[TABLE]

So E.3 holds. Now for any $\beta\in\mathbb{R}^{p}$, let $\beta=V\delta+\tilde{V}\tilde{\delta}$, then $\beta\in\ker(D_{S^{c}})$ if and only if $U_{S^{c}}\Lambda\delta=0$, which means $\delta\in\Lambda^{-1}\ker(U_{S^{c}})=\mathrm{Im}(\Lambda^{-1}U_{S}^{T}\tilde{U}^{\prime})$. So

[TABLE]

Since $\tilde{V}^{T}V=0$, the linear subspaces spanned by $J$ and $\tilde{V}$ are orthogonal, and we have

[TABLE]

Noting $\tilde{V}^{T}V=0,\ \tilde{V}^{T}X^{*}(I-U_{1}U_{1}^{T})=0$, we have

[TABLE]

Since $(XWW^{T}X^{*})^{{\dagger}}(XWW^{T}X^{*})$ is the projection matrix onto the linear subspace $\mathrm{Im}(XW)=\mathrm{Im}(X\tilde{V})+\mathrm{Im}(XJ)=\mathrm{Im}(U_{1})+\mathrm{Im}(XJ)$, and $(I-U_{1}U_{1}^{T})XJ=XJ-U_{1}\cdot U_{1}^{T}XJ$ lies in this subspace, the last term above becomes $X^{*}\left(I-U_{1}U_{1}^{T}\right)XJ$. Therefore, we get

[TABLE]

Now to prove E.2, it suffices to show

[TABLE]

which is surely true since $\tilde{U}^{\prime T}U^{\prime}=0$. Then $\mathrm{irr}(0)=\mathrm{ic}_{0}$ is proved.

Now we turn to $\mathrm{irr}(\infty)$. Let $M=U^{\prime\prime}\Lambda^{\prime\prime}U^{\prime\prime T}$ be the compact eigendecomposition of $M$, and $(U^{\prime\prime},\tilde{U}^{\prime\prime})$ is an orthogonal square matrix. Then

[TABLE]

when $\nu\rightarrow+\infty$. Besides, $\nu\Sigma_{S,S}\rightarrow I-U_{S}\tilde{U}^{\prime\prime}\tilde{U}^{\prime\prime T}U_{S}^{T}$, and this limit $\succeq\nu\Sigma_{S,S}\succ 0$ for any $\nu>0$. Thus $\Sigma_{S^{c},S}\Sigma_{S,S}^{-1}$ has limit when $\nu\rightarrow+\infty$.

Now we study when $\mathrm{irr}(\infty)=0$. Let

[TABLE]

Then $0=\tilde{V}^{T}D_{S}^{T}=\tilde{V}^{T}X^{T}C_{1}+0=\sqrt{n}V_{1}\Lambda_{1}U_{1}^{T}C_{1}$, which implies $U_{1}^{T}C_{1}=0$. So for $N=\Lambda^{-1}V^{T}X^{T}(I-U_{1}U_{1}^{T})/\sqrt{n}$, we have

[TABLE]

Then $\mathrm{irr}(\infty)=0\Longleftrightarrow-U_{S^{c}}\tilde{U}^{\prime\prime}\tilde{U}^{\prime\prime T}U_{S}^{T}=0\Longleftrightarrow-U_{S^{c}}(I-MM^{{\dagger}})U_{S}^{T}=0$. By $M=NN^{T}$, the equation is further equivalent to

[TABLE]

It suffices to show that the last property holds if and only if $\ker(X)\subseteq\ker(D_{S})$ or, equivalently, $\mathrm{Im}(D_{S}^{T})\subseteq\mathrm{Im}(X^{T})$. In fact, if $\mathrm{Im}(D_{S}^{T})\subseteq\mathrm{Im}(X^{T})$, then $C_{2}$ can be set [math] in the beginning, and $\mathrm{Im}(U_{S^{c}}^{T}C_{2})=\mathrm{Im}(0)\subseteq\mathrm{Im}(N)$. If $\mathrm{Im}(U_{S^{c}}^{T}C_{2})\subseteq\mathrm{Im}(N)$, let $U_{S^{c}}^{T}C_{2}=NC_{3}$, then

[TABLE]

and hence $D_{S}^{T}=X^{T}C_{1}+D_{S^{c}}^{T}C_{2}=X^{T}(C_{1}+(I-U_{1}U_{1}^{T})C_{3}/\sqrt{n})$, which implies $\mathrm{Im}(D_{S}^{T})\subseteq\mathrm{Im}(X^{T})$. We have finished the proof of that $\mathrm{irr}(\infty)=0$ if and only if $\ker(X)\subseteq\ker(D_{S})$.

Appendix F Proof on Oracle Properties

Proof of Lemma 1.

From the definition of oracle estimators 5.2,

[TABLE]

Adding F.1 to 5.1b and 5.1c, we have

[TABLE]

Besides, since

[TABLE]

by 5.2 and Pythagorean Theorem,

[TABLE]

where

[TABLE]

Noting $\gamma_{j}(t)\cdot\dot{\rho}_{j}(t)\equiv 0$ for each $j$, by 5.1d, F.2 and F.4 we have

[TABLE]

Thus it suffices to show

[TABLE]

Since $\|\gamma_{S}^{o}\|_{1}-\langle\gamma_{S}^{o},\rho_{S}^{\prime}(t)\rangle=0$ if $\|\gamma_{S}^{\prime}(t)-\gamma_{S}^{o}\|_{2}^{2}<(\gamma_{\min}^{o})^{2}$, and

[TABLE]

Thus

[TABLE]

It suffice to show

[TABLE]

which is true since by 1

[TABLE]

and by $F(\cdot+x)\geq F(\cdot)+x/(2\kappa)$

[TABLE]

Lemma 6**.**

Under 1, let $\gamma_{\min}^{o}:=\min(|\gamma_{j}^{o}|:\ \gamma_{j}^{o}\neq 0)$. For

[TABLE]

we have

[TABLE]

For $t\geq 0$, we have

[TABLE]

Proof of Lemma 6.

Noting F.3 and that $\ell(\beta^{\prime}(t),\gamma^{\prime}(t))$ is non-increasing, we know $L(t)$ is non-increasing. F.5 tells that $\Psi(t)$ is non-increasing since $L(t)\geq 0$. If $L(t)=0$ for $t=\tau_{\infty}(\mu)$, by F.6 and the fact that $L(t)$ is non-increasing, we have

[TABLE]

Therefore F.8 holds for $t\geq\tau_{\infty}(\mu)$. Now assume that $L(t)>0$ for $t=\tau_{\infty}(\mu)$ (and hence for $0\leq t\leq\tau_{\infty}(\mu)$), then $\Psi(t)$ is strictly decreasing on $[0,\tau_{\infty}(\mu)]$. Besides, $F$ is strictly increasing and continuous on $[(\gamma_{\min}^{o})^{2},+\infty)$. Moreover,

[TABLE]

If there does not exist some $t\leq\tau_{\infty}(\mu)$ satisfying F.8, then for $0\leq t\leq\tau_{\infty}(\mu)$,

[TABLE]

which also implies that $F^{-1}(\Psi(t))>0$. By Lemma 1,

[TABLE]

contradicting with the definition of $\tau_{\infty}(\mu)$. Thus F.8 holds for some $0\leq\tau\leq\tau_{\infty}(\mu)$. If $\kappa=+\infty$, we see that for $t\geq\tau_{\infty}(\mu)$, $\Psi(t)\leq\Psi(\tau)=0$. Then $-2L(t)$, the derivative of $\Psi(t)$, is [math] (which means $d(t)=0$) when $t\geq\tau_{\infty}(\mu)$, and F.8 holds. If $\kappa<+\infty$, just note that for $t\geq\tau$,

[TABLE]

So F.8 holds for $t\geq\tau_{\infty}(\mu)$.

For any $t>0$, if $L(t)=0$, then $d(t)=0$ and F.9 holds. If $L(t)>0$, let $C=\sqrt{2L(t)/\lambda_{H}}>0$, then for any $0\leq t^{\prime}\leq t$,

[TABLE]

Besides, for $\tilde{F}(x)=x/(2\kappa)+2\sqrt{sx}\geq F(x)$, by Lemma 1 we have

[TABLE]

By F.5 and the fact that

[TABLE]

we have that, if $d(0)>C$, then

[TABLE]

If $d(0)\leq C$, then similarly

[TABLE]

Combining it with F.6, we have

[TABLE]

Besides, noting C.3, we have

[TABLE]

Thus

[TABLE]

Thus F.9 holds.

Appendix G Proof on Consistency of Split LBISS

Before proving Theorem 4 and 5, we need the following lemmas.

Lemma 7** (No-false-positive condition for Split LBISS).**

For the oracle dynamics 5.1, if there is $\tau>0$, such that for $0\leq t\leq\tau$ the inequality

[TABLE]

*holds, then the solution path of the original dynamics 1.7 has no false-positive for $0\leq t\leq\tau$. *

Proof of Lemma 7.

It is easy to see that

[TABLE]

Now define the exit time of oracle subspace,

[TABLE]

It suffices to show $\tau_{\mathrm{exit}}>\tau$. For $0\leq t<\tau_{\mathrm{exit}}$, we have $\gamma_{S^{c}}(t)=0$, which also implies the paths of Split LBISS and oracle dynamics are identical, i.e. $\rho_{S}(t)=\rho_{S}^{\prime}(t)$ and $\gamma_{S}(t)=\gamma_{S}^{\prime}(t)$. Hence by G.2 we have

[TABLE]

We claim that

[TABLE]

(the equality above will be shown at last), so by G.3 we have

[TABLE]

Integration on both sides leads to, for $0\leq t<\tau_{\mathrm{exit}}$

[TABLE]

Due to the continuity of $\rho_{S^{c}}(t),\rho_{S}^{\prime}(t)$ (and $\gamma_{S}^{\prime}(t)$, if $\kappa<+\infty$), the equation above also holds for $t=\tau_{\mathrm{exit}}$. According to the definition of $\tau_{\mathrm{exit}}$, we know G.1 does not hold for $t=\tau_{\mathrm{exit}}$. Thus for $\tau<\tau_{\mathrm{exit}}$, the desired result follows.

So it suffices to prove

[TABLE]

Actually, let $H_{(\beta,S),(\beta,S)}=U^{\prime}\Lambda^{\prime}U^{\prime T}$ where $U^{\prime T}U^{\prime}=I$ and $\Lambda^{\prime}$ is an invertible diagonal matrix. It suffices to show $L\oplus\mathbb{R}^{s}=\mathrm{Im}(U^{\prime})$. First, by the definition of $H$, one can easily verify that

[TABLE]

On the other hand, assume that $(U^{\prime},\tilde{U}^{\prime})$ is an orthogonal square matrix. For any $\zeta\in L\oplus\mathbb{R}^{s}$, since $P_{\mathrm{Im}(U^{\prime})}\zeta\in\mathrm{Im}(U^{\prime})\subseteq L\oplus\mathbb{R}^{s}$, we have $P_{\mathrm{Im}(\tilde{U}^{\prime})}\zeta=\zeta-P_{\mathrm{Im}(U^{\prime})}\zeta\in L\oplus\mathbb{R}^{s}$, and 2.3 tells us

[TABLE]

Thus G.4 holds.

Lemma 8**.**

Suppose $\Sigma_{S,S}\succeq\lambda_{\Sigma}I$. For $\beta^{o}\in L$ and $\gamma_{S}^{o}\in\mathbb{R}^{s}$ satisfying F.1, we have

[TABLE]

and

[TABLE]

Besides, we have

[TABLE]

Proof.

By Lemma 4 and $\beta^{o}-\beta^{\star}\in L$, we have G.5. By F.1, we have

[TABLE]

and

[TABLE]

i.e.

[TABLE]

Left multiplying $\Lambda_{1}^{-2}V_{1}^{T}\tilde{V}^{T}$ on both sides of G.10 leads to

[TABLE]

Then left multiplying $V^{T}$ on both sides of G.10 leads to

[TABLE]

Recalling the definition of $B$ in Lemma 5, the equation above implies

[TABLE]

Plugging it into G.9, we obtain $\gamma_{S}^{o}-\gamma_{S}^{\star}=B_{\gamma}\epsilon$. Then noting $B\succeq\lambda_{D}^{2}I$, we have

[TABLE]

so G.8 holds. Now by G.12 we have $\delta^{o}-\delta^{\star}=B_{\delta}\epsilon$. Noting B.11 and $\Sigma_{S,S}\succeq\lambda_{\Sigma}I$, we have

[TABLE]

Thus

[TABLE]

which immediately leads to G.6. Finally, combining G.11 with G.6 we have G.7.

Now we are ready for proving the main theorems.

Proof of Theorem 4.

By B.3, G.8 and G.7, we have that with probability not less than $1-4s/m^{2}\geq 1-4/m$,

[TABLE]

By B.4, G.5, G.6, G.7 and G.8, with probability not less than $1-3\exp(-4n/5)$,

[TABLE]

The inequalities above also imply

[TABLE]

and

[TABLE]

From now, we assume all the inequalities above hold. The condition on $\kappa$ now tells us

[TABLE]

Now we prove the No-false-positive property. By Lemma 7, it suffices to show that for $0\leq t\leq\bar{\tau}$, G.1 holds with probability not less that $1-2/m$. By D.4, B.10 and F.9,

[TABLE]

Besides, by D.4 we have

[TABLE]

By B.11, $DA^{{\dagger}}D^{T}=U\Lambda B^{-1}\Lambda U^{T}$ and $\Lambda^{2}\preceq B\preceq(1+\nu\Lambda_{X}^{2}/\lambda_{D}^{2})\Lambda^{2}$, therefore $1$ is an upper bound of the largest eigenvalue of $DA^{{\dagger}}D^{T}$, and $1/(1+\nu\Lambda_{X}^{2}/\lambda_{D}^{2})$ is a lower bound of the smallest nonzero eigenvalue of $DA^{{\dagger}}D^{T}$. Then

[TABLE]

By B.3, with probability not less than $1-2/m$, for any $0\leq t\leq\bar{\tau}$,

[TABLE]

Combining the results above with 2, we have for $0\leq t\leq\bar{\tau}$, G.1 holds with probability not less that $1-2/m$, and we have the No-false-positive property (which tells that $(\beta(t),\gamma_{S}(t))$ coincides with that of the oracle dynamics for $0\leq t\leq\bar{\tau}$).

Then we prove the sign consistency of $\gamma(t)$. If the $\gamma_{\min}^{*}$ condition 4.3 holds, by G.13,

[TABLE]

Thus $\mathrm{sign}(\gamma_{S}^{o})=\mathrm{sign}(\gamma_{S}^{\star})$, and

[TABLE]

By F.8, the sign consistency of $\gamma_{S}^{\prime}(t)$ holds for

[TABLE]

thus also for $\bar{\tau}$. Then under the No-false-positive property,

[TABLE]

and

[TABLE]

Now we prove the $\ell_{2}$ consistency of $\gamma(t)$. Under the No-false-positive property, for $0\leq t\leq\bar{\tau}$,

[TABLE]

Finally, we prove the $\ell_{2}$ consistency of $\beta(t)$. Under the No-false-positive property, for $0\leq t\leq\bar{\tau}$,

[TABLE]

By Lemma 8 (especially noting G.12), we have

[TABLE]

By B.3, with probability not less than $1-2/m$, we have

[TABLE]

In this case, combining the inequalities above with $d(t)\leq 5\sqrt{s}/(\lambda_{H}t)$, the desired result follows.

Proof of Theorem 5.

By the proof details of Theorem 4, we know that with probability not less than $1-6/m-3\exp(-4n/5)$, G.13, G.14, G.15, G.16 and G.17 hold, meanwhile the solution path has no false-positive for $0\leq t\leq\overline{\tau}$. From now, we assume that these properties are all valid.

First we prove the sign consistency of $\tilde{\beta}(t)$. If the $\gamma_{\min}^{\star}$ condition 4.3 holds, then by Theorem 4, $S(\bar{\tau})=S$ holds, and we have

[TABLE]

To prove $\mathrm{sign}(D_{S}\tilde{\beta}(\bar{\tau}))=\mathrm{sign}(D_{S}\beta^{\star})$, note that

[TABLE]

First, by G.18, $\kappa\geq d(0)\geq\left\|\gamma_{S}^{o}\right\|_{2}\geq\gamma_{\min}^{o}$, and

[TABLE]

By F.8, we have $d\left(\bar{\tau}\right)\leq\gamma_{\min}^{o}/(8\Lambda_{D})$, and thus

[TABLE]

Besides, by G.6, we have

[TABLE]

with

[TABLE]

By B.3, with probability not less than $1-2/m$,

[TABLE]

Finally, we note G.19. Then $\mathrm{sign}(D_{S}\tilde{\beta}(\bar{\tau}))=\mathrm{sign}(D_{S}\beta^{\star})$ holds, since

[TABLE]

Then we prove the $\ell_{2}$ consistency of $\tilde{\beta}(t)$. For any $0\leq t\leq\bar{\tau}$, $S(t)\subseteq S$, which implies $D_{S^{c}}\tilde{\beta}(t)=D_{S^{c}}\beta^{\star}=0$. Then

[TABLE]

The first and second term of the right hand side are respectively not greater than

[TABLE]

(here we use the fact that $D_{S(t)^{c}}P_{S(t)}=0$), and

[TABLE]

Noting F.9 and G.14, as well as applying the definition of $B_{\delta}$ in Lemma 8, now we only need to show that with probability not less than $1-2/m-2r^{\prime}/m^{2}$,

[TABLE]

which are both true, according to B.3, as well as G.6 which leads to

[TABLE]

and

[TABLE]

Appendix H Proof on Consistency of Split LBI

Proof of Theorem 6 and 7.

They are merely discrete versions of proofs of Theorem 4 and 5. In the proofs, Lemma 9 and 10 stated below are applied, instead of Lemma 1 and 6.

Specifically, one can define the oracle iteration of Split LBI as an oracle version of Split LBI 1.6 (with $S$ known and $\rho_{k,S^{c}},\gamma_{k,S^{c}}$ set to be [math]), resembling the idea of oracle dynamics of Split LBISS. Define

[TABLE]

Then we have

Lemma 9** (Discrete Generalized Bihari’s inequality).**

Under 1, suppose $\kappa\alpha\|H\|_{2}<2$ and $\lambda_{H}^{\prime}=\lambda_{H}(1-\kappa\alpha\|H\|_{2}/2)$. For all $k$ we have

[TABLE]

*where $\gamma_{\min}^{o},\ F(x),\ F^{-1}(x)$ are defined the same as in Lemma 1. *

Proof of Lemma 9.

The proof is almost a discrete version of the continuous case. The only non-trivial thing is to show that

[TABLE]

By 1.6, we have

[TABLE]

Noting $(\rho_{k+1,S}^{\prime}-\rho_{k,S}^{\prime})^{T}\gamma_{k+1,S}^{\prime}\geq 0$ and multiplying $(d_{k,\beta}^{T},d_{\gamma,k,S}^{T})$ on both sides, we have

[TABLE]

Thus

[TABLE]

Lemma 10**.**

Under 1, suppose $\kappa\alpha\|H\|_{2}<2$ and $\lambda_{H}^{\prime}=\lambda_{H}(1-\kappa\alpha\|H\|_{2}/2)$. Let

[TABLE]

Then for any $k$ such that

[TABLE]

we have

[TABLE]

For any $k$, we have

[TABLE]

Proof of Lemma 10.

The proof is almost a discrete version of the continuous case. The only non-trivial thing is described as follows. First, suppose there does not exist $k\leq\tau_{\infty}^{\prime}(\mu)/\alpha$ satisfying H.2, then for any $0\leq k\alpha\leq\tau_{\infty}^{\prime}(\mu)$, we have $\Psi_{k}>\mu^{2}(\gamma_{\min}^{o})^{2}/(2\kappa)$. Letting $k_{0}=0$, then $\Psi_{k_{0}}=\Psi_{0}\leq F(d_{0}^{2})$. Suppose that

[TABLE]

Then $k_{4}\alpha>\tau_{\infty}^{\prime}(\mu)$. Besides, by Lemma 9,

[TABLE]

Thus $\lambda_{H}^{\prime}(k_{4}-4)\alpha$ is not greater than

[TABLE]

By $(u-v)/u\leq\log(u/v)$ and $(\sqrt{u}-\sqrt{v})/u\leq 1/\sqrt{v}-1/\sqrt{u}$ for $u\geq v>0$, the quantity above is not greater than

[TABLE]

Therefore we get

[TABLE]

a contradiction with the definition of $\tau_{\infty}^{\prime}(\mu)$. So there exists some $k\leq\tau_{\infty}^{\prime}(\mu)/\alpha$ satisfying H.2. Then continue to imitate the proof in the continous version, we obtain H.2 for all $k\geq\tau_{\infty}^{\prime}(\mu)/\alpha$. The proof of H.3 follows the same spirit.