Learning task structure via sparsity grouped multitask learning

TL;DR

This paper introduces a joint optimization approach to recover task group structures and sparsity patterns in high-dimensional multitask learning, improving feature selection accuracy.

Contribution

It proposes a novel regularizer-based joint optimization framework for simultaneously recovering task groups and sparsity patterns in multitask learning.

Findings

Accurately recovers task group structures in experiments.

Enhances feature selection in high-dimensional settings.

Demonstrates effectiveness through extensive empirical validation.

Abstract

Sparse mapping has been a key methodology in many high-dimensional scientific problems. When multiple tasks share the set of relevant features, learning them jointly in a group drastically improves the quality of relevant feature selection. However, in practice this technique is used limitedly since such grouping information is usually hidden. In this paper, our goal is to recover the group structure on the sparsity patterns and leverage that information in the sparse learning. Toward this, we formulate a joint optimization problem in the task parameter and the group membership, by constructing an appropriate regularizer to encourage sparse learning as well as correct recovery of task groups. We further demonstrate that our proposed method recovers groups and the sparsity patterns in the task parameters accurately by extensive experiments.

Figures

Tables

Table 1. Table 1: Sample complexity estimates of recovering group memberships and weights using different approaches

	SG-MTL	Lasso	Single group	Kang et al. (2011)
Samples	$O\left(Ns\log d+ms\right)$	$O\left(ms\log d\right)$	$O\left(msN\log d\right)$	$O\left(s\left(m+md\right)\right)$

Table 2. Table 2: Synthetic datasets: (Upper table) Average MSE from 5 fold CV. (Lower table) Varying group sizes and the corresponding average MSE. For each method, lowest MSE is highlighted.

Dataset	STL	ClusMTL	Kang	SG-MTL	Fusion SG-MTL
set-1 (3 groups)	1.067	1.221	1.177	0.682	0.614
set-2 (5 groups)	1.004	1.825	0.729	0.136	0.130
Synthetic data-2 with 30% feature overlap across groups
	Number of groups, $N$
Method	2	4	$N^{\ast }$=5	6	10
ClusMTL	1.900	1.857	1.825	1.819	1.576
Kang et al. (2011)	0.156	0.634	0.729	0.958	1.289
SG-MTL	0.145	0.135	0.136	0.137	0.137
Fusion SG-MTL	0.142	0.139	0.130	0.137	0.137

Table 3. Table 3: QSAR prediction: average MSE and R 2 superscript 𝑅 2 R^{2} over 10 train:test splits with 100 examples per task in the training split (i.e n 𝑛 n =100). The standard deviation of MSE is also shown. For all group learning methods, we use N 𝑁 N =5.

	${\mu }_{MSE}$	${\sigma }_{mse}$	${\mu }_{R^2}$	${\sigma }_{R2}$
STL	0.811	0.02	0.223	0.01
AllTasks	0.908	0.01	0.092	0.01
ClusMTL	0.823	0.02	0.215	0.02
GOMTL	0.794	0.01	0.218	0.01
Kang	1.011	0.03	0.051	0.03
Fusion SG-MTL	0.752	0.01	0.265	0.01

Table 4. Table 5: TFBS prediction: average AUC-PR, with training data size of 200 examples, test data of ≈ \approx 1800 for number of groups N 𝑁 N =10.

Method	AUC-PR
STL	0.825
AllTasks	0.709
ClusMTL	0.841
GOMTL	0.749
Kang	0.792
Fusion SG-MTL	0.837

Table 5. Table 7: 15 tasks corresponding to the 15 drug targets (target name and description is shown) and their groupings. The color of the cell indicates the assigned cluster.

3A4: CYP P450 3A4 inhibition	CB1: binding to cannabinoid receptor 1
DPP4: inhibition of dipeptidyl peptidase 4	HIVINT: inhibition of HIV integrase in a cell based assay
HIVPROT: inhibition of HIV protease	LOGD: lipophilicity measured by HPLC method
OX1: inhibition of orexin 1 receptor	METAB: percent remaining after 30 min microsomal incubation
OX2: inhibition of orexin 2 receptor	NK1: inhibition of neurokinin1 receptor binding
PGP: transport by p-glycoprotein	PPB: human plasma protein binding
RATF: rat bioavailability	TDI: time dependent 3A4 inhibitions
THROMBIN : human thrombin inhibition

Equations

$$\underset{{\mathbf{w}}^{(t)}\in {\mathbb{R}}^d}{\mathrm{minimize}}\mathcal{L}({\mathbf{w}}^{(t)};{\mathcal{D}}_t)+\lambda \Vert {\mathbf{w}}^{(t)}{\Vert }_1$$

$$\displaystyle\operatorname*{minimize}_{\bm{w}^{(1)},\ldots,\bm{w}^{(m)}}\,\sum_{t=1}^{m}\mathcal{L}(\bm{w}^{(t)};\mathcal{D}_{t})+\sum_{g\in\mathcal{G}^{*}}\lambda_{g}\big{\|}\bm{W}_{g}\big{\|}_{1,2}$$

$$\underset{\mathbf{W},\mathbf{U}}{\mathrm{minimize}}$$

$$s.t.$$

$$\underset{\mathbf{W},\mathbf{U}}{\mathrm{minimize}}$$

$$s.t.$$

$$\underset{\mathbf{W},\mathbf{U}}{\mathrm{minimize}}$$

$$s.t.$$

$$\underset{\mathbf{W},\mathbf{U}}{\mathrm{minimize}}$$

$$\displaystyle+\mu\sum_{t<t^{\prime}}\big{\|}\bm{V}_{t}-\bm{V}_{t^{\prime}}\big{\|}_{\text{F}}^{2}$$

$$s.t.$$

$$\displaystyle\sum_{g\in\mathcal{G}}\lambda_{g}\Bigg{\{}u_{g,t}(w_{j}^{(t)})^{2}+\Bigg{.}2\bigg{(}\sum_{j^{\prime}\neq j}\sqrt{\sum_{t^{\prime}=1}^{m}u_{g,t^{\prime}}(w_{j^{\prime}}^{(t^{\prime})})^{2}}\bigg{)}\sqrt{\sum_{t^{\prime}=1}^{m}u_{g,t^{\prime}}(w_{j}^{(t^{\prime})})^{2}}\Bigg{.}\Bigg{\}}+C(j,t)$$

$$\displaystyle\sum_{g\in\mathcal{G}}\lambda_{g}u_{g,t}(w_{j}^{(t)})^{2}+2\sum_{g\in\mathcal{G}^{+}}\lambda_{g}\bigg{(}\sum_{j^{\prime}\neq j}\sqrt{\sum_{t^{\prime}=1}^{m}u_{g,t^{\prime}}(w_{j^{\prime}}^{(t^{\prime})})^{2}}\bigg{)}\sqrt{\sum_{t^{\prime}=1}^{m}u_{g,t^{\prime}}(w_{j}^{(t^{\prime})})^{2}}$$

$$\displaystyle+2\sum_{g\in\mathcal{G}^{0}}\lambda_{g}\bigg{(}\sum_{j^{\prime}\neq j}\sqrt{\sum_{t^{\prime}=1}^{m}u_{g,t^{\prime}}(w_{j^{\prime}}^{(t^{\prime})})^{2}}\bigg{)}\sqrt{u_{g,t}}\big{|}w_{j}^{(t)}\big{|}$$

$$R(\mathbf{U})=$$

$$\displaystyle\nabla_{u_{g,t}}R(\bm{U})=\lambda_{g}\Big{(}\sum_{j=1}^{d}\big{\|}\bm{W}_{j}\sqrt{\bm{U}_{g}}\big{\|}_{2}\Big{)}\Big{(}\sum_{j=1}^{d}\frac{(w_{j}^{(t)})^{2}}{\big{\|}\bm{W}_{j}\sqrt{\bm{U}_{g}}\big{\|}_{2}}\Big{)}.$$

$$\nabla \text{makebox}[0.0pt][l]\mathcal{L}:={\nabla }_{w_j^{(t)}}\mathcal{L}(w_j^{(t)})+2\sum _{g\in \mathcal{G}}{\lambda }_g{w}_j^{(t)}+$$

$$\displaystyle 2\sum_{g\in\mathcal{G}^{+}}\lambda_{g}\bigg{(}\sum_{j^{\prime}\neq j}\sqrt{\sum_{t^{\prime}=1}^{m}u_{g,t^{\prime}}(w_{j^{\prime}}^{(t^{\prime})})^{2}}\bigg{)}\frac{u_{g,t}w_{j}^{(t)}}{\sqrt{\sum_{t^{\prime}=1}^{m}u_{g,t^{\prime}}(w_{j}^{(t^{\prime})})^{2}}},$$

$$\displaystyle\bar{\lambda}:=2\sum_{g\in\mathcal{G}^{0}}\lambda_{g}\bigg{(}\sum_{j^{\prime}\neq j}\sqrt{\sum_{t^{\prime}=1}^{m}u_{g,t^{\prime}}(w_{j^{\prime}}^{(t^{\prime})})^{2}}\bigg{)}\sqrt{u_{g,t}}.$$

Full text

1 Introduction

Humans acquire knowledge and skills by categorizing the various problems/tasks encountered, recognizing how the tasks are related to each other and taking advantage of this organization when learning a new task. Statistical machine learning methods also benefit from exploiting such similarities in learning related problems. Multitask learning (MTL) (Caruana, 1997) is a paradigm of machine learning, encompassing learning algorithms that can share information among related tasks and help to perform those tasks together more efficiently than in isolation. These algorithms exploit task relatedness by various mechanisms. Some works enforce that parameters of various tasks are close to each other in some geometric sense (Evgeniou & Pontil, 2004; Maurer, 2006). Several works leverage the existence of a shared low dimensional subspace (Argyriou et al., 2008; Liu et al., 2009; Jalali et al., 2010; Chen et al., 2012) or manifold (Agarwal et al., 2010) that contains the task parameters. Some bayesian MTL methods assume the same prior on parameters of related tasks (Yu et al., 2005; Daumé III, 2009), while neural networks based methods share some hidden units (Baxter, 2000).

A key drawback of most MTL methods is that they assume that all tasks are equally related. Intuitively, learning unrelated tasks jointly may result in poor predictive models; i.e tasks should be coupled based on their relatedness. While the coupling of task parameters can sometimes be controlled via hyper-parameters, this is infeasible when learning several hundreds of tasks. Often, knowing the task relationships themselves is of interest to the application at hand. While these relationships might sometimes be derived from domain specific intuition (Kim & Xing, 2010; Widmer et al., 2010; Rao et al., 2013), they are either not always known apriori or are pre-defined based on the knowledge of $P(X)$ rather than $P(Y|X)$. We aim to automatically learn these task relationships, while simultaneously learning individual task parameters. This idea of jointly learning task groups and parameters has been explored in prior works. For instance Argyriou et al. (2008) learn a set of kernels, one per group of tasks and Jacob et al. (2009) cluster tasks based on similarity of task parameters. Others  (Zhang & Yeung, 2010; Gong et al., 2012) try to identify “outlier” tasks. Kumar & Daume (2012); Kang et al. (2011) assume that task parameters within a group lie in a shared low dimensional subspace. Zhang & Schneider (2010) use a matrix-normal regularization to capture task covariance and feature covariance between tasks and enforce sparsity on these covariance parameters and Fei & Huan (2013) use a similar objective with a structured regularizer. Their approach is however, not suited for high dimensional settings and they do not enforce any sparsity constraints on the task parameters matrix $W$. A Bayesian approach is proposed in Passos et al. (2012), where parameters are assumed to come from a nonparametric mixture of nonparametric factor analyzers.

Here, we explore the notion of shared sparsity as the structure connecting a group of related tasks. More concretely, we assume that tasks in a group all have similar relevant features or analogously, the same zeros in their parameter vectors. Sparsity inducing norms such as the $\ell_{1}$ norm capture the principle of parsimony, which is important to many real-world applications, and have enabled efficient learning in settings with high dimensional feature spaces and few examples, via algorithms like the Lasso (Tibshirani, 1996). When confronted by several tasks where sparsity is required, one modeling choice is for each task to have its’ own sparse parameter vector. At the other extreme is the possibility of enforcing shared sparsity on all tasks via a structured sparsity inducing norm such as $\ell_{1}/\ell_{2}$ on the task parameter matrix: $\|\bm{W}\|_{1,2}$ (Bach et al., 2011). 222Note: this cross-task structured sparsity is different from the Group Lasso (Yuan & Lin, 2006), which groups covariates within a task (${\displaystyle\min_{w\in\mathbb{R}^{d}}}\sum_{g}\|w_{g}\|$, where $w_{g}$ is a group of parameters). We choose to enforce sparsity at a group level by penalizing $\|\bm{W}_{g}\|_{1,2}$, where $\|\bm{W}_{g}\|$ is the parameter matrix for all tasks in group $g$, while learning group memberships of tasks.

To see why this structure is interesting and relevant, consider the problem of transcription factor (TF) binding prediction. TFs are proteins that bind to the DNA to regulate expression of nearby genes. The binding specificity of a TF to an arbitrary location on the DNA depends on the pattern/sequence of nucleic acids (A/C/G/T) at that location. These sequence preferences of TFs have some similarities among related TFs. Consider the task of predicting TF binding, given segments of DNA sequence (these are the examples), on which we have derived features such as $n$-grams (called $k$-mers) 333e.g.: GTAATTNC is an 8-mer (‘N’ represents a wild card). The feature space is very high dimensional and a small set of features typically capture the binding pattern for a single TF. Given several tasks, each representing one TF, one can see that the structure of the ideal parameter matrix is likely to be group sparse, where TFs in a group have similar binding patterns (i.e similar important features but with different weights). The applicability of task-group based sparsity is not limited to isolated applications, but desirable in problems involving billions of features, as is the case with web-scale information retrieval and in settings with few samples such as genome wide association studies involving millions of genetic markers over a few hundred patients, where only a few markers are relevant.

The main contributions of this work are:

• •

We present a new approach towards learning task group structure in a multitask learning setting that simultaneously learns both the task parameters $W$ and a clustering over the tasks.

• •

We define a regularizer that divides the set of tasks into groups such that all tasks within a group share the same sparsity structure. Though the ideal regularizer is discrete, we propose a relaxed version and we carefully make many choices that lead to a feasible alternating minimization based optimization strategy. We find that several alternate formulations result in substantially worse solutions.

• •

We evaluate our method through experiments on synthetic datasets and two interesting real-world problem settings. The first is a regression problem: QSAR, quantitative structure activity relationship prediction (see (Ma et al., 2015) for an overview) and the second is a classification problem important in the area of regulatory genomics: transcription factor binding prediction (described above). On synthetic data with known group structure, our method recovers the correct structure. On real data, we perform better than prior MTL group learning baselines.

1.1 Relation to prior work

Our work is most closely related to Kang et al. (2011), who assume that each group of tasks shares a latent subspace. They find groups so that $\|\bm{W}_{g}\|_{*}$ for each group $g$ is small, thereby enforcing sparsity in a transformed feature space. Another approach, GO-MTL (Kumar & Daume, 2012) is based on the same idea, with the exception that the latent subspace is shared among all tasks, and a low-rank decomposition of the parameter matrix $\bm{W}={\bm{L}}{\bm{S}}$ is learned. Subsequently, the coefficients matrix $S$ is clustered to obtain a grouping of tasks. Note that, learning group memberships is not the goal of their approach, but rather a post-processing step upon learning their model parameters.

To understand the distinction from prior work, consider the weight matrix $\bm{W}^{*}$ in Figure 4(a), which represents the true group sparsity structure that we wish to learn. While each task group has a low-rank structure (since $s$ of the $d$ features are non-zero, the rank of any $\bm{W}_{g}$ is bounded by $s$), it has an additional property that ($d-s$) features are zero or irrelevant for all tasks in this group. Our method is able to exploit this additional information to learn the correct sparsity pattern in the groups, while that of Kang et al. (2011) is unable to do so, as illustrated on this synthetic dataset in Figure 5 (details of this dataset are in Sec 5.1). Though Kang et al. (2011) recovers some of the block diagonal structure of $W$, there are many non-zero features which lead to an incorrect group structure. We present a further discussion on how our method is sample efficient as compared to Kang et al. (2011) for this structure of $\bm{W}$ in Sec 3.1.

We next present the setup and notation, and lead to our approach by starting with a straight-forward combinatorial objective and then make changes to it in multiple steps (Sec 2-4). At each step we explain the behaviour of the function to motivate the particular choices we made; present a high-level analysis of sample complexity for our method and competing methods. Finally we show experiments (Sec 5) on four datasets.

2 Setup and Motivation

We consider the standard setting of multi-task learning in particular where tasks in the same group share the sparsity patterns on parameters. Let $\{T_{1},\ldots,T_{m}\}$ be the set of $m$ tasks with training data $\mathcal{D}_{t}$ ($t$ = $1\ldots m$). Let the parameter vectors corresponding to each of the $m$ tasks be $\bm{w}^{(1)},\bm{w}^{(2)},\ldots,\bm{w}^{(m)}\in\mathbb{R}^{d}$, $d$ is the number of covariates/features. Let $\mathcal{L}(\cdot)$ be the loss function which, given $\mathcal{D}_{t}$ and $\bm{w}^{(t)}$ measures deviation of the predictions from the response. Our goal is to learn the task parameters where i) each $\bm{w}^{(t)}$ is assumed to be sparse so that the response of the task can be succinctly explained by a small set of features, and moreover ii) there is a partition $\mathcal{G}^{*}:=\{G_{1},G_{2},\ldots,G_{N}\}$ over tasks such that all tasks in the same group $G_{i}$ have the same sparsity patterns. Here $N$ is the total number of groups learned. If we learn every task independently, we solve $m$ independent optimization problems:

[TABLE]

where $\|\bm{w}^{(t)}\|_{1}$ encourages sparse estimation with regularization parameter $\lambda$. However, if $\mathcal{G}^{*}$ is given, jointly estimating all parameters together using a group regularizer (such as $\ell_{1}/\ell_{2}$ norm), is known to be more effective. This approach requires fewer samples to recover the sparsity patterns by sharing information across tasks in a group:

[TABLE]

where $\bm{W}_{g}\in\mathbb{R}^{d\times|g|}$, where $|g|$ is the number of tasks in the group $g$ and $\|\cdot\|_{1,2}$ is the sum of $\ell_{2}$ norms computed over row vectors. Say $t_{1},t_{2}\ldots t_{k}$ belong to group $g$, then \big{\|}\bm{W}_{g}\big{\|}_{1,2}:=\sum_{j=1}^{d}\sqrt{(w^{(t_{1})}_{j})^{2}+(w^{(t_{2})}_{j})^{2}+\ldots+(w^{(t_{k})}_{j})^{2}}. Here $w^{(t)}_{j}$ is the $j$-th entry of vector $\bm{w}^{(t)}$. Note that here we use $\ell_{2}$ norm for grouping, but any $\ell_{\alpha}$ norm $\alpha\geq 2$ is known to be effective.

We introduce a membership parameter $u_{g,t}$: $u_{g,t}=1$ if task $T_{t}$ is in a group $g$ and [math] otherwise. Since we are only allowing a hard membership without overlapping (though this assumption can be relaxed in future work), we should have exactly one active membership parameter for each task: $u_{g,t}=1$ for some $g\in\mathcal{G}$ and $u_{g^{\prime},t}=0$ for all other $g^{\prime}\in\mathcal{G}\setminus\{g\}$. For notational simplicity, we represent the group membership parameters for a group $g$ in the form of a matrix $\bm{U}_{g}$. This is a diagonal matrix where $\bm{U}_{g}:=\text{diag}(u_{g,1},u_{g,2},\ldots,u_{g,m})\in\{0,1\}^{m\times m}$. In other words, $[\bm{U}_{g}]_{ii}=u_{g,i}=1$ if task $T_{i}$ is in group $g$ and [math] otherwise. Now, incorporating $\bm{U}$ in (1), we can derive the optimization problem for learning the task parameters $\{\bm{w}^{(t)}\}_{t=1,\ldots,m}$ and $\bm{U}$ simultaneously as follows:

[TABLE]

where $\bm{W}\in\mathbb{R}^{d\times m}:=[\bm{w}^{(1)},\bm{w}^{(2)},\ldots,\bm{w}^{(m)}]$. Here $\bm{I}^{m\times m}$ is the $m\times m$ identity matrix. After solving this problem, $\bm{U}$ encodes which group the task $T_{t}$ belongs to. It turns out that this simple extension in (2) fails to correctly infer the group structure as it is biased towards finding a smaller number of groups. Figure 1 shows a toy example illustrating this. The following proposition generalizes this issue.

Proposition 1

Consider the problem of minimizing (2) with respect to $\bm{U}$ for a fixed $\widehat{\bm{W}}$. The assignment such that $\widehat{\bm{U}}_{g}=\bm{I}^{m\times m}$ for some $g\in\mathcal{G}$ and $\widehat{\bm{U}}_{g^{\prime}}={\bm{0}}^{m\times m}$ for all other $g^{\prime}\in\mathcal{G}\setminus\{g\}$, is a minimizer of (2).

Proof: Please refer to the appendix.

3 Learning Groups on Sparsity Patterns

In the previous section, we observed that the standard group norm is beneficial when the group structure $\mathcal{G}^{*}$ is known but not suitable for inferring it. This is mainly because it is basically aggregating groups via the $\ell_{1}$ norm; let ${\bm{v}}\in\mathbb{R}^{N}$ be a vector of $(\|\bm{W}\bm{U}_{1}\|_{1,2},\|\bm{W}\bm{U}_{2}\|_{1,2},\ldots,\|\bm{W}\bm{U}_{N}\|_{1,2})^{\top}$, then the regularizer of (2) can be understood as $\|{\bm{v}}\|_{1}$. By the basic property of $\ell_{1}$ norm, $\bm{v}$ tends to be a sparse vector, making $\bm{U}$ have a small number of active groups (we say some group $g$ is active if there exists a task $T_{t}$ such that $u_{g,t}=1$.)

Based on this finding, we propose to use the $\ell_{\alpha}$ norm ($\alpha\geq 2$) for summing up the regularizers from different groups, so that the final regularizer as a whole forces most of $\|\bm{W}\bm{U}_{g}\|_{1,2}$ to be non-zeros:

[TABLE]

Note that strictly speaking, $\|{\bm{v}}\|_{\alpha}$ is defined as $(\sum_{i=1}^{N}|{v}_{i}|^{\alpha})^{1/\alpha}$, but we ignore the relative effect of $1/\alpha$ in the exponent. One might want to get this exponent back especially when $\alpha$ is large. $\ell_{\alpha}$ norms give rise to exactly the opposite effects in distributions of $u_{g,t}$, as shown in Figure 2 and Proposition 2.

Proposition 2

Consider a minimizer $\widehat{\bm{U}}$ of (3), for any fixed $\widehat{\bm{W}}$. Suppose that there exist two tasks in a single group such that $\widehat{w}^{(s)}_{i}\widehat{w}^{(t)}_{j}\neq\widehat{w}^{(s)}_{j}\widehat{w}^{(t)}_{i}$. Then there is no empty group $g$ such that $\widehat{\bm{U}}_{g}={\bm{0}}^{m\times m}$.

Proof: Please refer to the appendix.

Figure 3 visualizes the unit surfaces of different regularizers derived from (3) (i.e. \sum_{g\in\mathcal{G}}(\big{\|}\bm{W}\bm{U}_{g}\big{\|}_{1,2})^{\alpha}=1 for different choices of $\alpha$.) for the case where we have two groups, each of which has a single task. It shows that a large norm value on one group (in this example, on $G_{2}$ when $a$ = 0.9) does not force the other group (i.e $G_{1}$) to have a small norm as $\alpha$ becomes larger. This is evidenced in the bottom two rows of the third column (compare it with how $\ell_{1}$ behaves in the top row). In other words, we see the benefits of using $\alpha\geq 2$ to encourage more active groups.

While the constraint $[\bm{U}_{g}]_{ik}\in\{0,1\}$ in (3) ensures hard group memberships, solving it requires integer programming which is intractable in general. Therefore, we relax the constraint on $\bm{U}$ to $\quad 0\leq[\bm{U}_{g}]_{ik}\leq 1$. However, this relaxation along with the $\ell_{\alpha}$ norm over groups prevents both $\|\bm{W}\bm{U}_{g}\|_{1,2}$ and also individual $[\bm{U}_{g}]_{ik}$ from being zero. For example, suppose that we have two tasks (in $\mathbb{R}^{2}$) in a single group, and $\alpha=2$. Then, the regularizer for any $g$ can be written as \Big{(}\sqrt{(w^{(1)}_{1})^{2}\ u_{g,1}^{2}+(w^{(2)}_{1})^{2}\ u_{g,2}^{2}}+\sqrt{(w^{(1)}_{2})^{2}\ u_{g,1}^{2}+(w^{(2)}_{2})^{2}\ u_{g,2}^{2}}\Big{)}^{2}. To simply the situation, assume further that all entries of $\bm{W}$ are uniformly a constant $w$. Then, this regularizer for a single $g$ would be simply reduced to $4w^{2}(u_{g,1}^{2}+u_{g,2}^{2})$, and therefore the regularizer over all groups would be $4w^{2}(\sum_{t=1}^{m}\sum_{g}u_{g,t}^{2})$. Now it is clearly seen that the regularizer has an effect of grouping over the group membership vector $(u_{g_{1},t},u_{g_{2},t},\ldots,u_{g_{N},t})$ and encouraging the set of membership parameters for each task to be uniform.

To alleviate this challenge, we re-parameterize $u_{g,t}$ with a new membership parameter $u^{\prime}_{g,t}:=\sqrt{u_{g,t}}$. The constraint does not change with this re-parameterization: $0\leq u^{\prime}_{g,t}\leq 1$. Then, in the previous example, the regularization over all groups would be (with some constant factor) the sum of $\ell_{1}$ norms, $\|(u_{g_{1},t},u_{g_{2},t},\ldots,u_{g_{N},t})\|_{1}$ over all tasks, which forces them to be sparse. Note that even with this change, the activations of groups are not sparse since the sum over groups is still done by the $\ell_{2}$ norm.

Toward this, we finally introduce the following problem to jointly estimate $\bm{U}$ and $\bm{W}$ (specifically with focus on the case when $\alpha$ is set to $2$):

[TABLE]

where $\sqrt{\bm{M}}$ for a matrix $\bm{M}$ is obtained from element-wise square root operations of $\bm{M}$. Note that (3) and (3) are not equivalent, but the minimizer $\widehat{\bm{U}}$ given any fixed $\bm{W}$ usually is actually binary or has the same objective with some other binary $\widehat{\bm{U}}^{\prime}$ (see Theorem 1 of Kang et al. (2011) for details). As a toy example, we show in Figure 4, the estimated $\bm{U}$ (for a known $\bm{W}$) via different problems presented so far ((2), (3)).

Fusing the Group Assignments

The approach derived so far works well when the number of groups $N<<m$, but can create many singleton groups when $N$ is very large. We add a final modification to our objective to encourage tasks to have similar group membership wherever warranted. This makes the method more robust to the mis-specification of the number of groups, ‘$N$’ as it prevents the grouping from becoming too fragmented when $N>>N^{*}$. For each task $t=1,\ldots,m$, we define $N\times N$ matrix $\bm{V}_{t}:=\text{diag}(u_{1,t},\ldots,u_{N,t}).$ Note that the $\bm{V}_{t}$ are entirely determined by the $\bm{U}_{g}$ matrices, so no actual additional variables are introduced. Equipped with this additional notation, we obtain the following objective where $\|\cdot\|_{\text{F}}$ denotes the Frobenius norm (the element-wise $\ell_{2}$ norm), and $\mu$ is the additional regularization parameter that controls the number of active groups.

[TABLE]

3.1 Theoretical comparison of approaches:

It is natural to ask whether enforcing the shared sparsity structure, when groups are unknown leads to any efficiency in the number of samples required for learning. In this section, we will use intuitions from the high-dimensional statistics literature in order to compare the sample requirements of different alternatives such as independent lasso or the approach of  Kang et al. (2011). Since the formal analysis of each method requires making different assumptions on the data $X$ and the noise, we will instead stay intentionally informal in this section, and contrast the number of samples each approach would require, assuming that the desired structural conditions on the $x$’s are met. We evaluate all the methods under an idealized setting where the structural assumptions on the parameter matrix $W$ motivating our objective (2) hold exactly. That is, the parameters form $N$ groups, with the weights in each group taking non-zero values only on a common subset of features of size at most $s$. We begin with our approach.

Complexity of Sparsity Grouped MTL

Let us consider the simplest inefficient version of our method, a generalization of subset selection for Lasso which searches over all feature subsets of size $s$. It picks one subset $S_{g}$ for each group $g$ and then estimates the weights on $S_{g}$ independently for each task in group $g$. By a simple union bound argument, we expect this method to find the right support sets, as well as good parameter values in $O(Ns\log d+ms)$ samples. This is the complexity of selecting the right subset out of $d\choose s$ possibilities for each group, followed by the estimation of $s$ weights for each task. We note that there is no direct interaction between $m$ and $d$ in this bound.

Complexity of independent lasso per task

An alternative approach is to estimate an $s$-sparse parameter vector for each task independently. Using standard bounds for $\ell_{1}$ regularization (or subset selection), this requires $O(s\log d)$ samples per task, meaning $O(ms\log d)$ samples overall. We note the multiplicative interaction between $m$ and $\log d$ here.

Complexity of learning all tasks jointly

A different extreme would be to put all the tasks in one group, and enforce shared sparsity structure across them using $\|\cdot\|_{1,2}$ regularization on the entire weight matrix. The complexity of this approach depends on the sparsity of the union of all tasks which is $Ns$, much larger than the sparsity of individual groups. Since each task requires to estimate its own parameters on this shared sparse basis, we end up requiring $O(msN\log d)$ samples, with a large penalty for ignoring the group structure entirely.

Complexity of Kang et al. (2011)

As yet another baseline, we observe that an $s$-sparse weight matrix is also naturally low-rank with rank at most $s$. Consequently, the weight matrix for each group has rank at most $s$, plausibly making this setting a good fit for the approach of Kang et al. (2011). However, appealing to the standard results for low-rank matrix estimation (see e.g. Negahban & Wainwright (2011)), learning a $d\times n_{g}$ weight matrix of rank at most $s$ requires $O(s(n_{g}+d))$ samples, where $n_{g}$ is the number of tasks in the group $g$. Adding up across tasks, we find that this approach requires a total of $O(s(m+md))$, considerably higher than all other baselines even if the groups are already provided. It is easy to see why this is unavoidable too. Given a group, one requires $O(ms)$ samples to estimate the entries of the $s$ linearly independent rows. A method utilizing sparsity information knows that the rest of the columns are filled with zeros, but one that only knows that the matrix is low-rank assumes that the remaining $(d-s)$ rows all lie in the linear span of these $s$ rows, and the coefficients of that linear combination need to be estimated giving rise to the additional sample complexity. In a nutshell, this conveys that estimating a sparse matrix using low-rank regularizers is sample inefficient, an observation hardly surprising from the available results in high-dimensional statistics but important in comparison with the baseline of Kang et al. (2011).

For ease of reference, we collect all these results in Table 1 below.

4 Optimization

We solve (3) by alternating minimization: repeatedly solve one variable fixing the other until convergence (Algorithm 1) We discuss details below.

Solving (3) w.r.t $\bm{U}$: This step is challenging since we lose convexity due the reparameterization with a square root. The solver might stop with a premature $\bm{U}$ stuck in a local optimum. However, in practice, we can utilize the random search technique to get the minimum value over multiple re-trials. Our experimental results reveal that the following projected gradient descent method performs well.

Given a fixed $\bm{W}$, solving for $\bm{U}$ only involves the regularization term i.e R(\bm{U})=\sum_{g\in\mathcal{G}}\lambda_{g}\Big{(}\sum_{j=1}^{d}\big{\|}\bm{W}_{j}\sqrt{\bm{U}_{g}}\big{\|}_{2}\Big{)}^{2} which is differentiable w.r.t $U$. The derivative is shown in the appendix along with the extension for the fusion penalty from (3). Finally after the gradient descent step, we project $(u_{g_{1},t},u_{g_{2},t},\ldots,u_{g_{N},t})$ onto the simplex (independently repeat the projection for each task) to satisfy the constraints on it. Note that projecting a vector onto a simplex can be done in $O(m\log m)$ (Chen & Ye, 2011).

Solving (3) w.r.t $\bm{W}$

This step is more amenable in the sense that (3) is convex in $\bm{W}$ given $\bm{U}$. However, it is not trivial to efficiently handle the complicated regularization terms. Contrast to $\bm{U}$ which is bounded by $[0,1]$, $\bm{W}$ is usually unbounded which is problematic since the regularizer is (not always, but under some complicated conditions discovered below) non-differentiable at [math].

While it is challenging to directly solve with respect to the entire $\bm{W}$, we found out that the coordinate descent (in terms of each element in $\bm{W}$) has a particularly simple structured regularization.

Consider any $w_{j}^{(t)}$ fixing all others in $\bm{W}$ and $\bm{U}$; the regularizer $R(\bm{U})$ from (3) can be written as

[TABLE]

where $w_{j}^{(t)}$ is the only variable in the optimization problem, and $C(j,t)$ is the sum of other terms in (3) that are constants with respect to $w_{j}^{(t)}$.

For notational simplicity, we define $\kappa_{g,t}:=\sum_{t^{\prime}\neq t}u_{g,t^{\prime}}(w_{j}^{(t^{\prime})})^{2}$ that is considered as a constant in (6) given $\bm{U}$ and $\bm{W}\setminus\{w_{j}^{(t)}\}$. Given $\kappa_{g,t}$ for all $g\in\mathcal{G}$, we also define $\mathcal{G}^{0}$ as the set of groups such that $\kappa_{g,t}=0$ and $\mathcal{G}^{+}$ for groups s.t. $\kappa_{g,t}>0$. Armed with this notation and with the fact that $\sqrt{x^{2}}=|x|$, we are able to rewrite (6) as

[TABLE]

where we suppress the constant term $C(j,t)$. Since $\sqrt{x^{2}+a}$ is differentiable in $x$ for any constant $a>0$, the first two terms in (7) are differentiable with respect to $w_{j}^{(t)}$, and the only non-differentiable term involves the absolute value of the variable, $|w_{j}^{(t)}|$. As a result, (7) can be efficiently solved by proximal gradient descent followed by an element-wise soft thresholding. Please see appendix for the gradient computation of $\mathcal{L}$ and soft-thresholding details.

5 Experiments

We conduct experiments on two synthetic and two real datasets and compare with the following approaches.

1. Single task learning (STL): Independent models for each task using elastic-net regression/classification.

2. AllTasks: We combine data from all tasks into a single task and learn an elastic-net model on it

3. Clus-MTL: We first learn STL for each task, and then cluster the task parameters using k-means clustering. For each task cluster, we then train a multitask lasso model.

4. GO-MTL: group-overlap MTL (Kumar & Daume, 2012)

5. Kang et al. (2011): nuclear norm based task grouping

6. SG-MTL : our approach from equation 3

7. Fusion SG-MTL: our model with a fusion penalty (see Section 3)

5.1 Results on synthetic data

The first setting is similar to the synthetic data settings used in Kang et al. (2011) except for how $W$ is generated (see Fig 4(a) for our parameter matrix and compare it with Sec 4.1 of Kang et al. (2011)). We have 30 tasks forming 3 groups with 21 features and 15 examples per task. Each group in $\bm{W}$ is generated by first fixing the zero components and then setting the non-zero parts to a random vector $w$ with unit variance. $Y_{t}$ for task $t$ is $X_{t}W_{t}+\epsilon$. For the second dataset, we generate parameters in a similar manner as above, but with 30 tasks forming 5 groups, 100 examples per task, 150 features and a 30% overlap in the features across groups. In table 2, we show 5-fold CV results and in Fig 6 we show the groups ($\bm{U}$) found by our method.

How many groups? Table 2 (Lower) shows the effect of increasing group size on three methods (smallest MSE is highlighted). For our methods, we observe a dip in MSE when $N$ is close to $N^{*}$. In particular, our method with the fusion penalty gets the lowest MSE at $N^{*}=5$. Interestingly, Kang et al. (2011) seems to prefer the smallest number of clusters, possibly due to the low-rank structural assumption of their approach, and hence cannot be used to learn the number of clusters in a sparsity based setting.

5.2 Quantitative Structure Activity Relationships (QSAR) Prediction: Merck dataset

Given features generated from the chemical structures of candidate drugs, the goal is to predict their molecular activity (a real number) with the target. This dataset from Kaggle consists of 15 molecular activity data sets, each corresponding to a different target, giving us 15 tasks. There are between 1500 to 40000 examples and $\approx$ 5000 features per task, out of which 3000 features are common to all tasks. We create 10 train:test splits with 100 examples in training set (to represent a setting where $n\ll d$) and remaining in the test set. We report $R^{2}$ and MSE aggregated over these experiments in Table 4, with the number of task clusters $N$ set to 5 (for the baseline methods we tried $N$=2,5,7). We found that Clus-MTL tends to put all tasks in the same cluster for any value of $N$. Our method has the lowest average MSE.

In Figure 4, for our method we show how MSE changes with the number of groups $N$ (along x-axis) and over different sizes of the training/test split. The dip in MSE for $n=50$ training examples (purple curve marked ‘x’) around $N=5$ suggests there are 5 groups. The learned groups are shown in the appendix in Fig 7 followed by a discussion of the groupings.

5.3 Transcription Factor Binding Site Prediction (TFBS)

This dataset was constructed from processed ChIP-seq data for 37 transcription factors (TFs) downloaded from ENCODE database (Consortium et al., 2012). Training data is generated in a manner similar to prior literature (Setty & Leslie, 2015). Positive examples consist of ‘peaks’ or regions of the DNA with binding events and negatives are regions away from the peaks and called ‘flanks’. Each of the 37 TFs represents a task. We generate all 8-mer features and select 3000 of these based on their frequency in the data. There are $\approx$2000 examples per task, which we divide into train:test splits using 200 examples (100 positive, 100 negative) as training data and the rest as test data. We report AUC-PR averaged over 5 random train:test splits in Table 6. For our method, we found the number of clusters giving the best AUC-PR to be $N=10$. For the other methods, we tried $N$=5,10,15 and report the best AUC-PR.

Though our method does marginally better (not statistically significant) than the STL baseline, which is a ridge regression model, in many biological applications such as this, it is desirable to have an interpretable model that can produce biological insights. Our MTL approach learns groupings over the TFs which are shown in Fig 6(a). Overall, ClusMTL has the best AUC-PR on this dataset however it groups too many tasks into a single cluster (Fig 6(b)) and forces each group to have at least one task. Note how our method leaves some groups empty (column 5 and 7) as our objective provides a trade-off between adding groups and making groups cohesive.

6 Conclusion

We presented a method to learn group structure in multitask learning problems, where the task relationships are unknown. The resulting non-convex problem is optimized by applying the alternating minimization strategy. We evaluate our method through experiments on both synthetic and real-world data. On synthetic data with known group structure, our method outperforms the baselines in recovering them. On real data, we obtain a better performance while learning intuitive groupings. Code is available at: https://github.com/meghana-kshirsagar/treemtl/tree/groups

Full paper with appendix is available at: https://arxiv.org/abs/1705.04886

Acknowledgements We thank Alekh Agarwal for helpful discussions regarding Section 3.1. E.Y. acknowledges the support of MSIP/NRF (National Research Foundation of Korea) via NRF-2016R1A5A1012966 and MSIP/IITP (Institute for Information & Communications Technology Promotion of Korea) via ICT R&D program 2016-0-00563, 2017-0-00537.

Appendix 0.A Proof of proposition 1

Proof

Suppose some tasks $\widehat{\bm{U}}_{s}$ and $\widehat{\bm{U}}_{t}$ in different groups share at least one nonzero patterns. Then, it can be trivially shown that \sum_{g\in\mathcal{G}}\lambda_{g}\big{\|}\bm{W}\bm{U}_{g}\big{\|}_{1,2} term will decrease if we combine these two groups because $|a|+|b|>\sqrt{a^{2}+b^{2}}$ for any nonzero real number $a$ and $b$. Since $\widehat{\bm{U}}$ is fixed, the overall loss will decrease as well.

Appendix 0.B Proof of Proposition 2

Proof

We show the simplest case when $\alpha=2$. Suppose such tasks $\widehat{\bm{U}}_{s}$ and $\widehat{\bm{U}}_{t}$ in the statement. Then, it can be trivially shown that \sum_{g\in\mathcal{G}}\lambda_{g}\big{\|}\bm{W}\bm{U}_{g}\big{\|}_{1,2} term will decrease if we split these two tasks in different groups (we can change the group index either task $s$ or $t$ into empty group) because (|\widehat{w}^{(s)}_{i}|+|\widehat{w}^{(s)}_{j}|)^{2}+(|\widehat{w}^{(t)}_{i}|+|\widehat{w}^{(t)}_{j}|)^{2}<\Big{(}\sqrt{(\widehat{w}^{(s)}_{i})^{2}+(\widehat{w}^{(t)}_{i})^{2}}+\sqrt{(\widehat{w}^{(s)}_{j})^{2}+(\widehat{w}^{(t)}_{j})^{2}}\Big{)}^{2} unless $\widehat{w}^{(s)}_{i}\widehat{w}^{(t)}_{j}\neq\widehat{w}^{(s)}_{j}\widehat{w}^{(t)}_{i}$. Since $\widehat{\bm{U}}$ is fixed, the overall loss will decrease as well.

Appendix 0.C Gradient of regularizer terms from (3)

For $j=1,2,\ldots,d$, we define a row vector $\bm{W}_{j}\in\mathbb{R}^{1\times m}$ to denote the $j$-th row of the parameter matrix $\bm{W}$: $\bm{W}_{j}=(w^{(1)}_{j},w^{(2)}_{j},\ldots,w^{(m)}_{j})$. Then, with the definition of $\|\cdot\|_{1,2}$, we can rewrite the regularization terms:

[TABLE]

Recalling that that $\sqrt{\bm{U}_{g}}=\text{diag}(\sqrt{u_{g,1}},\sqrt{u_{g,2}},\ldots,\allowbreak\sqrt{u_{g,m}})$ where $u_{g,t}\in[0,1]$ and for any fixed task $T_{t}$, $\sum_{g\in\mathcal{G}}u_{g,t}=1$, the gradient with respect to $u_{g,t}$ can be computed as

[TABLE]

The gradient computation of (3) with the fusion term can be trivially extended since the squared Frobenius norm is uniformly differentiable; only the additional $2\sum_{t^{\prime}\neq t}(u_{g,t}-u_{g,t^{\prime}})$ term needs to be added in $\nabla_{u_{g,t}}R(\bm{U})$ above.

Appendix 0.D Optimization: gradient of $\mathcal{L}$ and soft-thresholding

(7) can be efficiently solved by proximal gradient descent followed by an element-wise soft thresholding with:

[TABLE]

followed by an element-wise soft thresholding $\mathcal{S}_{\nu}(a):=\text{sign}(a)\max(|a|-\nu,0)$. The amount of soft-thresholding is determined by the learning rate $\eta$ and the constant factor of $|w_{j}^{(t)}|$ which we call $\bar{\lambda}$:

[TABLE]

Appendix 0.E Interpretation of clusters learned

0.E.1 QSAR task clusters

For the results in Table 4, the number of task clusters $N$ is set to 5 (for the baseline methods we tried $N$=2,5,7). We found that Clus-MTL tends to put all tasks in the same cluster for any value of $N$. Our method has the lowest average MSE. In Figure 4, for our method we show how MSE changes with the number of groups $N$ (along x-axis) and over different sizes of the training/test split. The dip in MSE for $n=300$ (green box curve) between $N=5$ to $N=7$ suggests there are around 5-7 groups.

In Fig 7 we show the grouping of the 15 targets. Please note that since we do not use all features available for each task (we only selected the common 3000 features among all 15 tasks), the clustering we get is likely to be approximate. We show the clustering corresponding to a run with Avg MSE of 0.6082 (which corresponds to the dip in the MSE curve from Fig 4). We find that CB1, OX1, OX2, NK1 which are all receptors corresponding to neural pathways, are put in the same cluster (darker blue). OX1, OX2 are targets for sleep disorders, cocaine addiction. NK1 is a G protein coupled receptor (GPCR) found in the central nervous system and peripheral nervous system and has been targetted for controlling nausea and vomiting. CB1 is also a GPCR involved in a variety of physiological processes including appetite, pain-sensation, mood, and memory. In addition to these, DPP4 is also part of the same cluster; this protein is expressed on the surface of most cell types and is associated with immune regulation, signal transduction and apoptosis. Inhibitors of this protein have been used to control blood glucose levels. LOGD is also part of this cluster: lipophilicity is a general physicochemical property that determines binding ability of drugs. We are not sure what this task represents.

PGP and PPB are related to plasma and are in the same group. Thrombin, a protease and a blood-coagulant protein, HIV protease and 3A4 (which is involved in toxin removal) are in the same group. Further analysis of the features might reveal the commonality between mechanisms used to target these proteins.