Multitask Hopfield Networks

TL;DR

This paper introduces HoMTask, a novel multitask Hopfield Network model that leverages task sharing to improve classification accuracy, demonstrating competitive results with existing semi-supervised graph algorithms.

Contribution

First multitask model based on Hopfield Networks that effectively integrates multiple tasks into a single energy-based framework.

Findings

HoMTask improves classification over single-task Hopfield Networks.

Achieves competitive performance with state-of-the-art semi-supervised algorithms.

Provides theoretical guarantees on model coherence and convergence.

Abstract

Multitask algorithms typically use task similarity information as a bias to speed up and improve the performance of learning processes. Tasks are learned jointly, sharing information across them, in order to construct models more accurate than those learned separately over single tasks. In this contribution, we present the first multitask model, to our knowledge, based on Hopfield Networks (HNs), named HoMTask. We show that by appropriately building a unique HN embedding all tasks, a more robust and effective classification model can be learned. HoMTask is a transductive semi-supervised parametric HN, that minimizes an energy function extended to all nodes and to all tasks under study. We provide theoretical evidence that the optimal parameters automatically estimated by HoMTask make coherent the model itself with the prior knowledge (connection weights and node labels). The convergence properties of HNs are preserved, and the fixed point reached by the network dynamics gives rise to the prediction of unlabeled nodes. The proposed model improves the classification abilities of singletask HNs on a preliminary benchmark comparison, and achieves competitive performance with state-of-the-art semi-supervised graph-based algorithms.

Tables

Table 1. Table 1: Performance averaged across CC terms for different configuration of the model.

Configuration	AUC	AUPR
Branch, ${\sigma }_1$	0.961	0.439
Card, ${\sigma }_1$	0.959	0.439
Card, ${\sigma }_1$, $\alpha =0$	0.959	0.431
Card, ${\sigma }_1$, $\beta =0$	0.810	0.204
Card, ${\sigma }_1$, $\alpha =\beta =0$	0.811	0.204
Card, ${\sigma }_2$	0.937	0.312

Table 2. Table 2: Performance comparison averaged across GO branches. In bold the top results, underlined when statistically different than the second top result.

	RW	RWR	GBA	LP	MTLP	MTLP-inv	MS-kNN	RANKS	COSNet	HoMTask
AUC
CC	0.954	0.966	0.944	0.964	0.957	0.964	0.790	0.958	0.904	0.959
MF	0.934	0.955	0.931	0.951	0.939	0.953	0.742	0.945	0.859	0.945
BP	0.943	0.959	0.935	0.955	0.947	0.961	0.764	0.949	0.855	0.954
AUPR
CC	0.367	0.437	0.207	0.308	0.343	0.342	0.218	0.398	0.361	0.439
MF	0.199	0.272	0.125	0.201	0.229	0.234	0.090	0.236	0.214	0.291
BP	0.244	0.313	0.145	0.224	0.246	0.250	0.116	0.271	0.241	0.330

Equations

$$\small{\scriptsize x_{i}(t)=}\left\{\begin{array}[]{rl}\sin\rho&\mbox{\hskip 8.5359ptif\ }\overset{i-1}{\underset{j=1}{\sum}}w_{ij}x_{j}(t)+\overset{n}{\underset{k=i+1}{\sum}}w_{ik}x_{k}(t-1)-\lambda>0\\ -\cos\rho&\mbox{\hskip 8.5359ptif\ }\overset{i-1}{\underset{j=1}{\sum}}w_{ij}x_{j}(t)+\overset{n}{\underset{k=i+1}{\sum}}w_{ik}x_{k}(t-1)-\lambda\leq 0\\ \end{array}\right.$$

$$E(\mathbf{x})=-\frac{1}{2}\sum _{i\ne j}{w}_{ij}{x}_i{x}_j+\lambda \sum _{i=1}^n{x}_i.$$

$$\Vert {\mathbf{x}}^{(k)}-{\mathbf{x}}^{(r)}{\Vert }^2,$$

$$E_{\mathcal{H}}(\boldsymbol{X})=\sum_{k=1}^{m}\left(E\big{(}\boldsymbol{x}^{(k)}\big{)}+\frac{\alpha}{4}\sum_{\begin{subarray}{c}r=1\\ r\neq k\end{subarray}}^{m}s_{kr}\ \|\boldsymbol{x}^{(k)}-\boldsymbol{x}^{(r)}\|^{2}\right),$$

$$\Vert {\mathbf{x}}^{(k)}-{\mathbf{x}}^{(r)}{\Vert }^2=\Vert {\mathbf{x}}^{(k)}{\Vert }^2+\Vert {\mathbf{x}}^{(r)}{\Vert }^2-2{\mathbf{x}}^{(k)}\cdot {\mathbf{x}}^{(r)},$$

$$\sum_{k=1}^{m}\sum_{\begin{subarray}{c}r=1\\ r\neq k\end{subarray}}^{m}s_{kr}\big{(}\|\boldsymbol{x}^{(k)}\|^{2}+\|\boldsymbol{x}^{(r)}\|^{2}\big{)}=2\Big{(}\sum_{\begin{subarray}{c}k=1\end{subarray}}^{m}S_{k}\|\boldsymbol{x}^{(k)}\|^{2}\Big{)}\ ,$$

$$E_{\mathcal{H}}(\boldsymbol{X})=\sum_{k=1}^{m}\left(E\big{(}\boldsymbol{x}^{(k)}\big{)}+\frac{\alpha}{2}\Big{(}S_{k}\sum_{i=1}^{n}x_{ik}^{2}-\sum_{\begin{subarray}{c}r=1\\ r\neq k\end{subarray}}^{m}s_{kr}\sum_{i=1}^{n}x_{ik}x_{ir}\Big{)}\right).$$

$$x_{ik}(t+1)=\{\begin{array}{ll}\sin {\rho }_k,& \text{mbox}if{\varphi }_{ik}(t)>0\\ -\cos {\rho }_k,& \text{mbox}if{\varphi }_{ik}(t)\le 0\end{array}$$

$${\varphi }_{ik}(t):=A_{ik}(t)-{\theta }_{ik}+\alpha B_{ik}(t)$$

$$\begin{array}{rl}{E}_{ik}(t)=-\frac{1}{2}{x}_{ik}(t)\sum _{j=1}^h(w_{ij}+w_{ji})x_{jk}(t)+{\gamma }_k{x}_{ik}+& \frac{\alpha S_k}{2}{x}_{ik}^2-\\ & \frac{\alpha }{2}{x}_{ik}\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^m(s_{kr}+s_{rk})x_{ir}.\end{array}$$

$$\begin{split}\textstyle\Delta_{ik}E(t+1)=-\big{(}x_{ik}&(t+1)-x_{ik}(t)\big{)}\\ &\Big{(}A_{ik}(t)-\gamma_{k}-\frac{\alpha S_{k}}{2}\big{(}x_{ik}(t+1)+x_{ik}(t)\big{)}+\alpha B_{ik}(t)\Big{)}.\end{split}$$

$$E_{\mathcal{H_{L}}}(\boldsymbol{L})=\sum_{k=1}^{m}\left(E_{L}\big{(}\boldsymbol{l}^{(k)}\big{)}+\frac{\alpha}{2}\Big{(}S_{k}\sum_{i\in L}l_{ik}^{2}-\sum_{\begin{subarray}{c}r=1\\ r\neq k\end{subarray}}^{m}s_{kr}\sum_{i\in L}l_{ik}l_{ir}\Big{)}\right)\ ,$$

$${\bar{l}}_{ik}=\{\begin{array}{ll}\sin {\rho }_k,& \text{mbox}ifi\in L_{k,+}\\ -\cos {\rho }_k,& \text{mbox}ifi\in L_{k,-}\end{array},$$

$$\begin{split}\phi^{L}_{ik}(\boldsymbol{\gamma},\boldsymbol{\rho})=\sum_{j\in L}w_{ij}\Big{(}\sin\rho_{k}\chi_{jk}-\cos\rho_{k}&\big{(}1-\chi_{jk}\big{)}\Big{)}-\beta_{k}+\\ &\alpha\sum_{\begin{subarray}{c}r=1\\ r\neq k\end{subarray}}s_{kr}\Big{(}\sin\rho_{k}\chi_{ir}-\cos\rho_{k}\big{(}1-\chi_{ir}\big{)}\Big{)}\ ,\end{split}$$

$$\begin{array}[]{l}\mathrm{TP}(i,k,\boldsymbol{\gamma},\boldsymbol{\rho})=f(\tau\phi_{ik}(\boldsymbol{\gamma},\boldsymbol{\rho})),\hskip 99.58464pti\in L_{k,+}\\[2.84544pt] \mathrm{FN}(i,k,\boldsymbol{\gamma},\boldsymbol{\rho})=1-f(\tau\phi_{ik}(\boldsymbol{\gamma},\boldsymbol{\rho})),\hskip 82.51282pti\in L_{k,+}\\[2.84544pt] \mathrm{FP}(i,k,\boldsymbol{\gamma},\boldsymbol{\rho})=f(\tau\phi_{ik}(\boldsymbol{\gamma},\boldsymbol{\rho})),\hskip 98.16191pti\in L_{k,-}\end{array}$$

$$\textstyle\mbox{{F}}(\boldsymbol{\gamma},\boldsymbol{\rho})=\sigma\big{(}F_{1}(\boldsymbol{\gamma},\boldsymbol{\rho}),F_{2}(\boldsymbol{\gamma},\boldsymbol{\rho}),\ldots,F_{m}(\boldsymbol{\gamma},\boldsymbol{\rho})\big{)}\ ,$$

$$F_k(\mathbf{\gamma },\mathbf{\rho })=\frac{2\sum _{i\in L_{k,+}}\mathrm{TP}(i,k,\mathbf{\gamma },\mathbf{\rho })}{2\sum _{i\in L_{k,+}}\mathrm{TP}(i,k,\mathbf{\gamma },\mathbf{\rho })+\sum _{i\in L_{k,-}}\mathrm{FP}(i,k,\mathbf{\gamma },\mathbf{\rho })+\sum _{i\in L_{k,+}}\mathrm{FN}(i,k,\mathbf{\gamma },\mathbf{\rho })}$$

$$F(\mathbf{\gamma },\mathbf{\rho })=1⟹F_1(\mathbf{\gamma },\mathbf{\rho })=F_2(\mathbf{\gamma },\mathbf{\rho })=\dots =F_m(\mathbf{\gamma },\mathbf{\rho })=1.$$

$$E_{\mathcal{H_{U}}}(\boldsymbol{U})=\sum_{k=1}^{m}\left(E_{U}\big{(}\boldsymbol{u}^{(k)}\big{)}+\frac{\alpha}{2}\Big{(}S_{k}\sum_{i=1}^{h}u_{ik}^{2}-\sum_{\begin{subarray}{c}r=1\\ r\neq k\end{subarray}}^{m}s_{kr}\sum_{i=1}^{h}u_{ik}u_{ir}\Big{)},\right)$$

$$u_{ik}(t+1)=\{\begin{array}{ll}\sin \hat{\rho },& \text{mbox}if{\varphi }_{ik}^U(t)>0\\ -\cos \hat{\rho },& \text{mbox}if{\varphi }_{ik}^U(t)\le 0\end{array},$$

$${\eta }_k{\left(\sum _{i=1}^h(a_k{u}_{ik}+b_k)-h{p}_{k,+}\right)}^2,$$

$$h{p}_{k,+}=\arg \underset{q}{\max }Prob\left\{|U_{k,+}|=q|L\text{mbox}contains|L_{k,+}|\text{mbox}positives\right\},$$

$$\eta_{k}a_{k}\Big{(}a_{k}\sum\limits_{i=1}^{h}\sum\limits_{\begin{subarray}{c}j=1\\ j\neq i\end{subarray}}^{h}u_{ik}u_{jk}+\big{(}2b_{k}(h-1)+1-2p_{k,+}\big{)}\sum\limits_{i=1}^{h}u_{ik}\Big{)},$$

$$E_{U}\big{(}\boldsymbol{u}^{(k)}\big{)}=-\frac{1}{2}\sum_{i=1}^{h}\sum_{\begin{subarray}{c}j=1\\ j\neq i\end{subarray}}^{h}w_{ij}^{(k)}u_{ik}u_{jk}+\sum_{i=1}^{h}u_{ik}\tilde{\theta}_{ik},$$

$$\begin{split}E_{\mathcal{H_{U}}}(\boldsymbol{U})=\sum_{k=1}^{m}\Bigg{(}-\frac{1}{2}{\boldsymbol{u}^{(k)}}^{T}\boldsymbol{W}^{(k)}_{\textrm{UU}}\boldsymbol{u}^{(k)}+&{\boldsymbol{u}^{(k)}}^{T}\tilde{\boldsymbol{\theta}}_{k}+\\ &\frac{\alpha}{2}\Big{(}S_{k}\sum\limits_{i=1}^{h}u_{ik}^{2}-\sum\limits_{\begin{subarray}{c}r=1\\ r\neq k\end{subarray}}^{m}s_{kr}\sum_{i=1}^{h}u_{ik}u_{ir}\Big{)}\Bigg{)}\end{split}$$

$$s_{kr}=\left\{\begin{array}[]{cl}{\displaystyle\frac{\big{|}L_{k,+}\wedge L_{r,+}\big{|}}{\big{|}L_{k,+}\vee L_{r,+}\big{|}}}&\text{if $L_{k,+}\vee L_{r,+}\neq\emptyset$}\\ 0&\text{otherwise.}\end{array}\right.$$

Full text

1 Introduction

Multitask learning is concerned with simultaneously learning multiple prediction tasks that are related to one another. It has been frequently observed in the recent literature that, when there are relations between the tasks, it can be advantageous to learn them simultaneously instead of learning each task separately [7, 11]. A major challenge in multitask learning is how to selectively screen the sharing of information so that unrelated tasks do not end up influencing each other. Sharing information between two unrelated tasks can worsen the performance of both tasks.

Multitasking thus plays an important role in a variety of practical situations, including: the prediction of user ratings for unseen items based on rating information from related users [32], the simultaneously forecasting of many related financial indicators [19], the categorization of genes associated with a genetic disorder by exploiting genes associated with related diseases [15].

There is a vast literature on multitask learning. The most important lines of work include: regularizers biasing the solution towards functions that lie geometrically close to each other in a RKHS [12, 11], or lie in a low dimensional subspace [3, 26]; structural risk minimization methods, where multitask relations are established by enforcing predictive functions for the different tasks to belong to the same hypothesis set [2]; spectral [10, 4] and cluster-based [23, 38] assumptions on the task relatedness; Bayesian approaches where task parameters share a common prior [39, 9, 42]; methods allowing a small number of outlier tasks that are not related to any other task [40, 8]; approaches attempting to learn the full task covariance matrix [41, 20]. To our knowledge, no multitask attempts have been proposed for Hopfield networks (HNs) [21], whereas several studies investigated HNs as singletask classifier [24, 27, 17, 22]. Indeed, HNs are efficient local optimizers, using the local minima of the energy function determined by network dynamics as a proxy to node classification.

In this paper we develop HoMTask, Hopfield multitask Network, an approach to multitask learning based on exploiting a family of parametric HNs. Our approach builds on COSNet [6], a singletask HN proposed to classify instances in a transductive semi-supervised scenario with unbalanced data. A main feature of HoMTask is that the energy function is extended to all tasks to be learned and to all instances (labeled and unlabeled), so as to learn the model parameters and to infer the node labels simultaneously for all tasks. The obtained network can be seen as a collection of singletask HNs, appropriately interconnected by exploiting the task relatedness. In particular, each task is associated with a couple of parameters determining the neuron activation values and thresholds, and we theoretically proved that in the optimal case, the learning procedure adopted is able to learn the parameters so as to move the multitask state of the labeled sub-network to a minimum of the energy. This is an important result, which allows the model to better fit the input data, since the classification of unlabeled nodes is based upon a minimum of the unlabeled subnetwork. Another interesting feature of HoMTask is that the complexity of the learning procedure linearly increases with the number of tasks, thus allowing the model to nicely scale on settings including numerous tasks. Finally, a proof of convergence of the multitask dynamics to a minimum of the energy is also supplied.

Experiments on a real-world classification problem have shown that HoMTask remarkably outperforms singletask HNs, and has competitive performance with state-of-the-art graph-based methods proposed in the same context.

2 Methods

2.1 Problem definition

The problem input is composed of an undirected weighted graph $G(V,\boldsymbol{W})$, where $V=\{1,2,\ldots,n\}$ is the set of instances and the non negative symmetric matrix $\boldsymbol{W}=\left(w_{ij}\right)$ denotes the degree of functional similarity between each pair of nodes $i$ and $j$. A set of binary learning tasks $C=\{c_{k}|k=1,2,\dots,m\}$ over $G$ is given, where for every task $c_{k}$, $V$ is labelled with $\{+,-\}$. The labeling is known only for the subset $L\subset V$, whereas it is unknown for $U:=V\setminus L$. Moreover, the subsets of vertices labelled with $+$ (positive) and $-$ (negative) are denoted by $L_{k,+}$ and $L_{k,-}$, respectively, for each task $c_{k}\in C$. Without loss of generality, we assume $U=\{1,2,\cdots,h\}$ and $L=\{h+1,h+2,\cdots,n\}$. As further assumption, task labelings are highly unbalanced, that is $\frac{|L_{k,+}|}{|L_{k,-}|}\ll 1$, for each $k\in\{1,2,\ldots,m\}$. In the multitask scenario, a $m\times m$ symmetric matrix $\boldsymbol{S}=s_{kr}|_{k,r=1}^{m}$ is also given, where $s_{kr}\in[0,1]$ is an index of relatedness/similarity between the tasks $c_{k}$ and $c_{r}$, and $s_{kk}=0$ for each $k\in\{1,2,\dots,m\}$.

The aim is determining a set of bipartitions $(U_{k,+},U_{k,-})$ of vertices in $U$ for each task $c_{k}\in C$ by jointly learning tasks in $C$, on the basis of the prior information encoded in $G$ and $\boldsymbol{S}$.

In the following, the bold font is adopted to denote vectors and matrices, and the calligraphic font to denote multitask Hopfield networks. Moreover, we denote by $\boldsymbol{W}_{LL}$ and $\boldsymbol{W}_{UU}$ the submatrices of $\boldsymbol{W}$ relative to nodes $L$ and $U$, respectively.

2.2 Previous singletask modelling

In this section we recall the basic model proposed in [6, 13] for singletask modeling, named COSNet, that has inspired the multitask setting presented here. Essentially, it relies on a parametric family of the Hopfield model [21], where the network parameters are learned to cope with the label imbalance and the network equilibrium point is interpreted to classify the unlabeled nodes. A COSNet network over $G=\langle V,\boldsymbol{W}\rangle$ is a the triple $H=\langle\boldsymbol{W},\lambda,\rho\rangle$, where $\lambda\in\mathbb{R}$ denotes the neuron activation threshold (unique for all neurons), and $\rho\in[0,\frac{\pi}{2})$ is a parameter which determines the two neuron activation (state) values $\{\sin\rho,-\cos\rho\}$. The model parameters are appropriately learned in order to allow the algorithm to counterbalance the large imbalance towards negatives (see [13]). The initial state of a neuron $i\in V$ is set to $x_{i}(0)=\sin\rho$, if $i$ is positive, $x_{i}(0)=-\cos\rho$, if $i$ is negative, and $x_{i}(0)=0$ when $i$ in unlabeled. The network evolves according to the following asynchronous dynamics:

[TABLE]

where $x_{i}(t)$ is the state of neuron $i\in V$ at time $t$. At each time $t$, the vector $\boldsymbol{x}(t)=(x_{1}(t),x_{2}(t),\ldots,x_{h}(t))$ represents the state of the whole network. The network admits a state function named energy function:

[TABLE]

The convergence properties of the dynamics (1) depend on the weight matrix structure $\boldsymbol{W}$ and the rule by which the nodes are updated. In particular, if the matrix is symmetric and the dynamic is asynchronous, it has been proved that the network converges to a stable state in polynomial time. As a major result, it has been shown that (2) is a Lyapunov function for the Hopfield dynamical systems with asynchronous dynamics, i.e., for each $t>0$, $E(\boldsymbol{x}(t+1))\leq E(\boldsymbol{x}(t))$ and exists a time $\bar{t}$ such that $E(\boldsymbol{x}(t))=E(\boldsymbol{x}(\bar{t}))$, for all $t\geq\bar{t}$. Moreover, the reached fixed point $\bar{\boldsymbol{x}}=\boldsymbol{x}(\bar{t})$ is a local minimum of (2). Then, a neuron $i$ in $U$ is classified as positive if $\bar{x}_{i}=\sin\rho$, as negative otherwise.

2.3 Multitask Hopfield networks

A Hopfield multitask network, named HoMTask, with neurons $V$ is a quadruple $\mathcal{H}=\langle\boldsymbol{W},\boldsymbol{\gamma},\boldsymbol{\rho},\boldsymbol{S}\rangle$, where $\boldsymbol{S}$ is the task similarity matrix, $\boldsymbol{\gamma}=(\gamma_{1},\ldots,\gamma_{m})\in\mathbb{R}^{m}$, $\boldsymbol{\rho}=(\rho_{1},\ldots,\rho_{m})\in[\frac{\pi}{4},\frac{\pi}{2})^{m}$. The couple of parameters $(\gamma_{k},\rho_{k})$ is associated with task $c_{k}$, for each $k\in\{1,2,\ldots,m\}$: by leveraging the approach adopted in COSNet, for a task $c_{k}$, the neuron activation values are $\{\sin\rho_{k},-\cos\rho_{k}\}$, whereas $\gamma_{k}$ is the neuron activation threshold (the same for every neuron). Such a formalization allows to keep the absolute activation value in the range $[0,1]$, and to calibrate it by suitably learning $\rho_{k}\in[\frac{\pi}{4},\frac{\pi}{2})$. For instance, in presence of a large majority of negative neurons, $\rho_{k}$ close to $\frac{\pi}{2}$ would prevent positive neurons to be overwhelmed during the net dynamics.

The state of the network is the $n\times m$ matrix $\boldsymbol{X}=(\boldsymbol{x}^{(1)},\boldsymbol{x}^{(2)},\ldots,\boldsymbol{x}^{(m)})$, where $\boldsymbol{x}^{(k)}=(x_{1k},\ x_{2k},\ \ldots,x_{nk})\in\{\sin\rho_{k},-\cos\rho_{k}\}^{n}$ is the state vector corresponding to task $c_{k}$. When simultaneously learning related tasks $c_{k}$ and $c_{r}$, an usual approach consists in expecting that the higher the relatedness $s_{rk}$, the closer the corresponding states. In our setting, this can be achieved by minimizing

[TABLE]

for any couple of tasks $c_{k},c_{r}\in C$, with $k\neq r$. To this end, we incorporate a term proportional to $\sum_{k}\sum_{r}s_{kr}\|\boldsymbol{x}^{(k)}-\boldsymbol{x}^{(r)}\|^{2}$ into the energy of $\mathcal{H}$, thus obtaining:

[TABLE]

where E\big{(}\boldsymbol{x}^{(k)}\big{)}=-\frac{1}{2}{\boldsymbol{x}^{(k)}}^{T}\boldsymbol{W}\boldsymbol{x}^{(k)}+{\boldsymbol{x}^{(k)}}^{T}\gamma_{k}\boldsymbol{e}_{n}, $\boldsymbol{e}_{n}$ is the $n$-dimensional vector made by all ones, and $\alpha$ is a real hyper-parameter regulating the multitask contribution. Without the second additive term in brackets, energy (3) would be the summation of the energy functions of $m$ independent singletask Hopfield networks, as recalled in the previous section.

By using the equalities

[TABLE]

where $\cdot$ denotes the inner product, and giving that

[TABLE]

with $S_{k}=\sum_{r=1}^{m}s_{kr}$, the energy (3) can be rewritten as:

[TABLE]

Informally, $\mathcal{H}$ can be thought as $m$ interconnected singletask parametric Hopfield networks $H_{1}=\langle\boldsymbol{W},\boldsymbol{\gamma}_{1},\boldsymbol{\rho}_{1}\rangle,\ldots,H_{m}=\langle\boldsymbol{W},\boldsymbol{\gamma}_{m},\boldsymbol{\rho}_{m}\rangle$ on $V$, having all the same topology given by $\boldsymbol{W}$. In addition, the multitask energy term introduces self loops for all neurons, and a novel connection for each neuron $i\in V$ with $i$ itself in the network $H_{r}$, $r\in C\setminus c_{k}$, whose weight is $\alpha s_{kr}$ (see Fig.1).

It is worth nothing that usually in Hopfield networks there are no self-loops; nevertheless, we show that it does not affect the convergence properties of the overall network.

2.3.1 Update rule and dynamics convergence.

Starting from an initial state $\boldsymbol{X}(0)$ and adopting the asynchronous dynamic, in $nm$ steps all neurons are updated in random order according to the following update rule:

[TABLE]

where $x_{ik}(t+1)$ is the state of neuron $i\in X$ in task $c_{k}$ ($ik$-th) at time $t+1$, and

[TABLE]

is the input of the $ik$-th neuron at time $t$, whose terms are $A_{ik}(t)=\sum\limits_{j=1}^{n}w_{ij}x_{jk}(t)$, \theta_{ik}=\gamma_{k}+\frac{\alpha S_{k}}{2}\big{(}\sin\rho_{k}-\cos\rho_{k}\big{)}, and $B_{ik}(t)=\sum\limits_{\begin{subarray}{c}r=1\\ r\neq k\end{subarray}}^{m}s_{kr}x_{ir}(t)$. $A_{ik}$ represents the singletask input (eq. 1), $B_{ik}(t)$ is the multitask contribution, and $\theta_{ik}$ is the activation threshold for neuron $ik$, including also the ‘singletask’ threshold. The form of $\theta_{ik}$ derives from the following theorem, stating a HoMTask Hopfield network preserves the convergence properties of a Hopfield network.

Theorem 2.1

A HoMTask Hopfield network $\mathcal{H}=\langle\boldsymbol{W},\boldsymbol{\gamma},\boldsymbol{\rho},\boldsymbol{S}\rangle$ with $n$ neurons and the asynchronous dynamics (5), which starts from any given network state, eventually reaches a stable state at a local minimum of the energy function (4).

Proof

Let $E_{ik}(t)$ be the energy contribution to (4) of the $ik$-th neuron at time $t$, with

[TABLE]

Let $\Delta_{ik}E(t+1)=E_{ik}(t+1)-E_{ik}(t)$ be the energy variation after updating the state $x_{ik}$ at time $t+1$ according to (5). Due to the symmetry of $\boldsymbol{W}$ and $\boldsymbol{S}$, it follows

[TABLE]

Since the energy (4) is lower bounded, to complete to proof we need to prove that after updating $x_{ik}$ at time $t+1$ according to (5), it holds $\Delta_{ik}E(t+1)\leq 0$. From (7), when $x_{ik}(t+1)=x_{ik}(t)$, that is when the neuron does not change state, it follows $\Delta_{ik}E(t+1)=0$. Accordingly, we need to investigate the remaining two cases: (a) $x_{ik}(t)=\sin\rho_{k}$ and $x_{ik}(t+1)=-\cos\rho_{k}$; (b) $x_{ik}(t)=-\cos\rho_{k}$ and $x_{ik}(t+1)=\sin\rho_{k}$. In both cases it holds (by definition of $\theta_{ik}$) \gamma_{k}-\frac{\alpha S_{k}}{2}\big{(}x_{ik}(t+1)+x_{ik}(t)\big{)}\ \begin{subarray}{c}=\end{subarray}\ \theta_{ik}.

1. (a)

$(x_{ik}(t+1)-x_{ik}(t))=(-\cos\rho_{k}-\sin\rho_{k})<0$, and, according to (5), $A_{ik}(t)-\theta_{ik}+\alpha B_{ik}(t)\leq 0$. It follows $\Delta_{ik}E(t+1)\leq 0$. 2. (b)

$(x_{ik}(t+1)-x_{ik}(t))=(\sin\rho_{k}+\cos\rho_{k})>0$, and $A_{ik}(t)-\theta_{ik}+\alpha B_{ik}(t)>0$. Thus $\Delta_{ik}E(t+1)<0$.

Every neuron update thereby does not increase the energy of the network, and, since the energy is lower bounded, there will be a time $t^{\prime}>0$ from which the update of any given neuron will not change the current state, which is the definition of equilibrium state of the network, and which makes $\boldsymbol{X}(t^{\prime})$ a local minimum of (4). ∎

2.3.2 Learning the model parameters.

Considered the subnetwork $\mathcal{H_{L}}=\langle\boldsymbol{W}_{\textrm{LL}},\boldsymbol{\gamma},\boldsymbol{\rho},\boldsymbol{S}\rangle$ restricted to labeled nodes $L$, its energy is:

[TABLE]

where $\boldsymbol{L}=(\boldsymbol{l}^{(1)},\boldsymbol{l}^{(2)},\ldots,\boldsymbol{l}^{(m)})$ with components $\boldsymbol{l}^{(k)}=(l_{1k},l_{2k},\dots,l_{(n-h)k})$ belonging to the set $\{\sin\rho_{k},-\cos\rho_{k}\}^{(n-h)}$, and E_{L}\big{(}\boldsymbol{l}^{(k)}\big{)}=-\frac{1}{2}{\boldsymbol{l}^{(k)}}^{T}\boldsymbol{W}_{\textrm{LL}}\boldsymbol{l}^{(k)}+{\boldsymbol{l}^{(k)}}\cdot\gamma_{k}\boldsymbol{e}_{(n-h)}.

The given bipartition $(L_{k,+},L_{k,-})$ for each task $c_{k}$ naturally induces the labeling $\bar{\boldsymbol{l}}^{(k)}=\{\bar{l}_{1k},\bar{l}_{2k},\ldots,\bar{l}_{(n-h)k}\}$, defined as it follows:

[TABLE]

and constituting the known ‘multitask’ state $\bar{\boldsymbol{L}}=(\bar{\boldsymbol{l}}^{(1)},\bar{\boldsymbol{l}}^{(2)},\ldots,\bar{\boldsymbol{l}}^{(m)})$.

Given $\bar{\boldsymbol{L}}$ as known components of a final state $\bar{\boldsymbol{X}}$ of the multitask network $\mathcal{H}=\langle\boldsymbol{W},\boldsymbol{\gamma},\boldsymbol{\rho},\boldsymbol{S}\rangle$, the purpose of the learning step is to compute the pair ($\hat{\boldsymbol{\gamma}}$, $\hat{\boldsymbol{\rho}}$) which makes $\bar{\boldsymbol{X}}$ an energy global minimizer of (3), the energy function associated with $\mathcal{H}$. Since our aim is also keeping the model scalable on large sized data, and finding the global minimum of the energy requires time/memory intensive procedures, we employ a learning procedure leading $\bar{\boldsymbol{L}}$ towards an fixed point of $\mathcal{H}_{L}$, being in general a local minimum of (8). We provide the details of the learning procedure in the following, showing that such an approach also helps to handle the label imbalance at each task.

Maximizing a cost-sensitive criterion.

When the parameters $\boldsymbol{\gamma},\boldsymbol{\rho}$ are fixed, each neuron $ik$ has input

[TABLE]

where, for each $k\in\{1,\dots,m\}$ and $j\in L$, $\chi_{jk}=1$ if $j\in L_{k,+}$, [math] otherwise. $\phi^{L}_{ik}$ corresponds to $\phi_{ik}$ of equation (6) restricted to $L$; to simplify the notation, in the following $\phi^{L}_{ik}$ is thereby denoted by $\phi_{ik}$. Since the subnetwork is labeled, it is possible to define the set of true positive $tp_{k}(\boldsymbol{\gamma},\boldsymbol{\rho})=\{i\in L_{k,+}|\phi_{ik}(\boldsymbol{\gamma},\boldsymbol{\rho})>0\}$, false negative $fn_{k}(\boldsymbol{\gamma},\boldsymbol{\rho})=\{i\in L_{k,+}|\phi_{ik}(\boldsymbol{\gamma},\boldsymbol{\rho})\leq 0\}$, and false positive $fp_{k}(\boldsymbol{\gamma},\boldsymbol{\rho})=\{i\in L_{k,-}|\phi_{ik}(\boldsymbol{\gamma},\boldsymbol{\rho})>0\}$, for every task $c_{k}$. Following the approach proposed in [16], a set of membership functions can be defined, extending the crisp memberships introduced above:

[TABLE]

where $f:\mathbb{R}\rightarrow[0,1]$ is a suitable monotonically increasing membership function. For instance f_{1}(x)=\nicefrac{{1}}{{\big{(}1+e^{(-x)}\big{)}}} or f_{2}(x)=\frac{1}{2}\big{(}\frac{2}{\pi}\operatorname{arctg}(x)+1\big{)}. $\tau>0$ is a real parameter. If $f$ is the Heaviside step function, we obtain the crisp memberships. For example, when $f=f_{1}$ or $f=f_{2}$, if $i\in L_{k,+}$ and $\tau\phi_{ik}(\boldsymbol{\gamma},\boldsymbol{\rho})=0$, if follows $\mathrm{TP}(i,k,\boldsymbol{\gamma},\boldsymbol{\rho})=\mathrm{FN}(i,k,\boldsymbol{\gamma},\boldsymbol{\rho})=0.5$; if $i\in L_{k,+}$ and $\tau\phi_{ik}(\boldsymbol{\gamma},\boldsymbol{\rho})\rightarrow\infty$, it follows $\mathrm{TP}(i,k,\boldsymbol{\gamma},\boldsymbol{\rho})=1$ and $\mathrm{FN}(i,k,\boldsymbol{\gamma},\boldsymbol{\rho})=0$. The intermediate cases lead to $0<\mathrm{TP}(i,k,\boldsymbol{\gamma},\boldsymbol{\rho}),\mathrm{FN}(i,k,\boldsymbol{\gamma},\boldsymbol{\rho})<1$.

Such a generalization, in a different setting (singletask, multi-category) increased both the learning capability of the model and its classification performance [16]. By means of the membership functions (9), we can define the objective $F$:

[TABLE]

where

[TABLE]

and $\sigma$ is an appropriately chosen function, e.g. the mean, the minimum, or the harmonic mean function. The property $\sigma$ must satisfy is that

[TABLE]

By definition, $F_{k}$ (a generalization of the F-measure) is penalized more by the misclassification of a positive instance than by the misclassification of a negative one. By maximizing $F(\boldsymbol{\gamma},\boldsymbol{\rho})$ we can thereby cope with the label imbalance. To this end, the learning criterion for the model parameters adopted here is $\displaystyle(\hat{\boldsymbol{\gamma}},\hat{\rho})=\arg\underset{\boldsymbol{\gamma},\boldsymbol{\rho}}{\max}\hskip 2.84544pt\mbox{{F}}(\boldsymbol{\gamma},\boldsymbol{\rho})$, which also leads to the following important result.

Theorem 2.2

If $\mbox{{F}}(\boldsymbol{\gamma},\boldsymbol{\rho})=1$, then $\bar{\boldsymbol{L}}$ is an equilibrium state of the sub-network $\mathcal{H_{L}}\langle W_{\textrm{LL}},\boldsymbol{\gamma},\boldsymbol{\rho},\boldsymbol{S}\rangle$.

Learning procedure.

Denoted by $\boldsymbol{\delta}=(\boldsymbol{\gamma},\boldsymbol{\rho})$ the vector of model parameters, this procedure learns the values $\hat{\boldsymbol{\delta}}$ that maximize eq. (10), that is \displaystyle\hat{\boldsymbol{\delta}}={\arg}\underset{\boldsymbol{\delta}}{\max}\hskip 2.84544pt\mbox{F}(\boldsymbol{\delta}). Following the approach proposed in [16], we adopt the simplest search method [28], which adopts an iterative and incremental procedure estimating in turn a single parameter at a time, by fixing the other ones, until a suitable criterion is met (e.g. convergence, number of iterations, etc.). It thereby allows the complexity of the learning procedure to increase linearly with the number of tasks. In particular, for a fixed assignment of parameters $(\delta_{1},\ldots,\delta_{i-1},\delta_{i+1},\ldots,\delta_{2m})$, we estimate $\hat{\delta}_{i}$ with the value \overline{\delta}_{i}=\operatorname{argmax}_{\delta_{i}}\hskip 2.84544pt\mbox{F}(\boldsymbol{\delta}), $i\in\{1,\ldots,2m\}$. The learning procedure is sketched below:

Randomly permute the vector $\boldsymbol{\delta}$, and randomly initialize $\boldsymbol{\delta}$; 2. 2.

Determine an estimate $\overline{\delta}_{i}$ of $\hat{\delta}_{i}$ with a standard line search procedure for optimizing continuous functions of one variable, and fix $\delta_{i}=\overline{\delta}_{i}$; 3. 3.

Iterate Step 2 for each $i\in\{1,2,\ldots,2m\}$; 4. 4.

Repeat Step 3 till a stopping criterion is satisfied.

As stopping criterion we used a combination of the maximum number of iterations and of the maximum norm of the difference of two subsequent estimates $\overline{\boldsymbol{\delta}}$ (falling below a given threshold). As initial test, at Step 2 we simply adopted a grid search optimization algorithm, where a set of trials is formed for each parameter, and all possible parameter combinations are assembled and tested.

2.3.3 Label inference.

Once the parameters $\hat{\boldsymbol{\gamma}},\hat{\boldsymbol{\rho}}$ have been estimated, we consider the subnetwork $\mathcal{H_{U}}=\langle\boldsymbol{W}_{\textrm{UU}},\hat{\boldsymbol{\gamma}},\hat{\boldsymbol{\rho}},\boldsymbol{S}\rangle$ restricted to the unlabeled nodes $U$, whose energy is

[TABLE]

with $\boldsymbol{U}=(\boldsymbol{u}^{(1)},\boldsymbol{u}^{(2)},\ldots,\boldsymbol{u}^{(m)})$ state of $\mathcal{H_{U}}$, $\boldsymbol{u}^{(k)}=(u_{1k},\ u_{2k},\ \ldots,u_{hk})=(x_{1k},\ x_{2k},\ \ldots,x_{hk})\in\{\sin\hat{\rho}_{k},-\cos\hat{\rho}_{k}\}^{h}$, E_{U}\big{(}\boldsymbol{u}^{(k)}\big{)}=-\frac{1}{2}{\boldsymbol{u}^{(k)}}^{T}\boldsymbol{W}_{UU}\boldsymbol{u}^{(k)}+{\boldsymbol{u}^{(k)}}^{T}\overline{\boldsymbol{\theta}}_{k}, and $\overline{\boldsymbol{\theta}}_{k}=\hat{\gamma}_{k}\boldsymbol{e}_{h}-W_{\textrm{UL}}\bar{\boldsymbol{l}}^{(k)}$ is the vector of activation thresholds for task $c_{k}$, including the contribution of labeled nodes (which are clamped).

In the case the learned parameters make $\bar{\boldsymbol{L}}$ a part of global minimum of $\mathcal{H}$, by determining the global minimum of $\mathcal{H_{U}}$, we can successfully determine the global minimum of $\mathcal{H}$ (as stated by the following theorem), and consequently the solution of the problem.

Theorem 2.3

Given a multitask Hopfield network $\mathcal{H}=\langle W,\boldsymbol{\gamma},\boldsymbol{\rho},\boldsymbol{S}\rangle$ on neurons $V$, bipartitioned into the sets $L$ and $U$, if $\boldsymbol{L}$ is a part of a global minimum of the energy of $\mathcal{H}$, and $\boldsymbol{U}$ is a global minimum of the energy of $\mathcal{H}_{U}=\langle W_{UU},\boldsymbol{\gamma},\boldsymbol{\rho},\boldsymbol{S}\rangle$, then $(\boldsymbol{L},\boldsymbol{U})$ is a global minimum of the energy of $\mathcal{H}$.

On the other side, computing the energy global minimum of $\mathcal{H}_{U}$ would require time intensive algorithms; accordingly, to preserve the model efficiency and scalability, we run the dynamics of $\mathcal{H}_{U}$ till an equilibrium state is reached, which, in general, is an energy local minimum.

Given an initial state $\boldsymbol{U}(0)$, at each time $t$ one neuron is updated, and in $nm$ consecutive steps all neurons are updated asynchronously and in a randomly chosen order according to the following update rule:

[TABLE]

where $u_{ik}(t+1)$ is the state of neuron $ik$ at time $t+1$, and $\phi^{U}_{ik}(t)$ is the restriction of $\phi_{ik}(t)$ to $U$. According to Theorem 2.1, the dynamics (12) converges to an equilibrium state $\bar{\boldsymbol{U}}$ of $\mathcal{H}_{U}$, and the predicted bipartition $(U_{k,+},U_{k,-})$ for task $k$ is: $U_{k,+}:=\{i\in U|\bar{u}_{ik}=\sin\hat{\rho}\}$ and $U_{k,-}:=\{i\in U|\bar{u}_{ik}=-\cos\hat{\rho}\}$.

2.3.4 Dynamics regularization.

As shown by [13], the network dynamics might get stuck in trivial equilibrium states when input labeling are highly unbalanced —e.g. states made up by almost all negative neurons. To prevent this behaviour, they applied a dynamics regularization, with the aim to control the number of positive neurons in the current state. By extending that approach, and denoted by $p_{k,+}=\frac{|L_{k,+}|}{|L|}$ the proportion of positives in the training set for task $c_{k}$, we add to the energy function $E_{\mathcal{H_{U}}}(\boldsymbol{U})$ the regularization term

[TABLE]

where $a_{k}=\frac{1}{\sin\hat{\rho}_{k}+\cos\hat{\rho}_{k}}$, $b_{k}=\frac{\cos\hat{\rho}_{k}}{\sin\hat{\rho}_{k}+\cos\hat{\rho}_{k}}$, and $\eta_{k}$ is a real regularization parameter. Since $a_{k}$ and $b_{k}$ are such that $(a_{k}u_{ik}+b_{k})=1$ when $u_{ik}=\sin\hat{\rho}_{k}$, [math] otherwise, the $\sum_{i=1}^{h}(a_{k}u_{i}+b_{k})$ is the number of positive neurons in $\boldsymbol{u}^{(k)}$. The term (13) is thereby minimized when the number of positive neurons in $\boldsymbol{u}^{(k)}$ is $hp_{k,+}$. This choice is motivated by the fact that

[TABLE]

when $U$ and $L$ are randomly drawn from $V$ —see [13]. By simplifying eq. (13), up to a constant terms, we obtain the quadratic term:

[TABLE]

which can be thereby included into $E_{U}(\boldsymbol{u}^{(k)})$:

[TABLE]

where $\tilde{\theta}_{ik}=\overline{\theta}_{ik}+\eta_{k}a_{k}\left[2b_{k}(h-1)+(1-2p_{k,+}h)\right]$ and $w_{ij}^{(k)}=(w_{ij}-2\eta_{k}a_{k}^{2})$. By adding a regularization term for each task $c_{k}$, we obtain the following overall energy:

[TABLE]

Informally, this regularization leads to a different network topology for each task, in addition to a modification of the neuron activation thresholds. Nevertheless, since the connection weights are modified by a constant value, from an implementation standpoint this regularization just need to memorize $m$ different constant values, thus not increasing the space complexity of the model. As preliminary approach, and to have a fair comparison, the parameters $\eta_{k}$ have been set as for the singletask case [13], that is \eta_{k}=\begin{array}[]{rl}\beta\big{|}\tan\big{(}(\hat{\rho}_{k}-\frac{\pi}{4})*2\big{)}\big{|}&\mbox{,}\end{array} where $\beta$ is a non negative real constant. Another advantage of this choice is that we have to learn just one parameter $\beta$, instead of $m$ dedicated parameters.

3 Preliminary results and discussion

In this section we evaluate our algorithm on the prediction of the bio-molecular functions of proteins, a binary classification problem aiming at associating sequenced proteins with their biological functions. In the following, we describe the experimental setting, analyze the impact on performance of parameter configurations, and finally we compare our algorithm against other state-of-the-art graph-based methods.

3.1 Benchmark data

In our experiments we considered the Gene Ontology [5] terms, i.e. the reference functional classes in this context, and their annotations to the Saccaromyces cerevisiae (yeast) proteins, one of the most studied model organisms. The connection matrix $\boldsymbol{W}$ has been retrieved from the STRING database, version 10.5 [35], and contains $6391$ yeast proteins. As common in this context, the GO terms with less than $10$ and more than $100$ yeast protein annotations (positives) have been discarded, in order to have a minimum of information and to avoid too generic terms —GO is a DAG, where annotations for a term are transferred to all its ancestors. We considered the UniProt GOA (release $87$, 12 March 2018) experimentally validated annotations from all GO branches, Cellular Component (CC), Molecular Function (MF) and Biological Process (BP), for a total of $162$, $227$, and $660$ CC, MF, BP GO terms, respectively.

3.2 Evaluation setting

To evaluate the generalization capabilities of our algorithm, we used a $3$-fold cross validation (CV), and measured the performance in terms of Area Under the ROC curve (AUC) and Area Under the Precision-Recall curve (AUPR). The AUPR has been adopted in the recent CAFA2 international challenge for the critical assessment of protein functions [25], since in this imbalanced setting AUPR is more informative than AUC [33].

3.3 Model configuration

HoMTask has three hyper-parameters, $\tau$, $\beta$ and $\alpha$, and two functions to be chosen: $f$ in eq. (9), and $\sigma$ in eq. (10). $\tau$, $\beta$ and $\alpha$ were learned through internal $3$-fold CV, considering also the cases $\alpha$ and $\beta$ in turn or together clamped to $=0$, to evaluate their individual impact on the performance. A different discussion can be made for the $\tau$ parameter, since in our experimentations best performance correspond to large values of $\tau$ (e.g. $\tau>500$), thus making the model less sensitive to this choice (the function $f$ becomes a Heaviside function). This behaviour apparently conflicts with results reported in [16], where typically $0.5<\tau<2$ performed best. However, in that work the authors focused on a substantially different learning task, i.e. a singletask Hopfield model, where nodes were divided into categories, and the model parameters were not related to different tasks, but to different node categories. We still include $\tau$ in the formalization proposed in Section 2.3 because it permits also future analytic studies about the derivatives of $\sigma$, to determine close formulations for the optimal parameters. Further, We set f(x)=\frac{1}{2}\big{(}\frac{2}{\pi}\operatorname{arctg}(x)+1\big{)}, since this choice in a multi-category context leaded to excellent results [16], even if different choices are possible (Section 2.3).

On the other side, we tested two choices for $\sigma$: the harmonic mean ($\sigma_{1}$) and mean functions ($\sigma_{2}$). Furthermore, another central factor of our model is the computation of the task similarity matrix $\boldsymbol{S}$, which can be computed by using several metrics (see for instance [15]), and how to group the tasks that should be learned together. We employed in this work the Jaccard similarity measure, since it performed nicely in hierarchical contexts [15, 37, 14], defined as follows:

[TABLE]

Thus, $s_{kr}$ is the ratio between the number of instances that are positive for both tasks and the number of instances that are positive for at least one task. The higher the number of shared instances, the higher the similarity (up to $1$); conversely, if two task do not share items, their similarity is zero.

Finally, we grouped tasks by GO branch, and by GO branch and number of positives: in the first case (Branch), all tasks within a GO branch are learned simultaneously; in the latter one (Card), tasks in the same branch having $10$-$20$, $21$-$50$, or $51$-$100$ positives have been grouped together. Both approaches are quite usual when predicting GO terms [31, 14].

Table 1 reports the obtained results on the $CC$ terms. First, the two different strategies for grouping tasks led to similar results in this setting, with the Branch grouping being experimentally slower because the learning procedure needs more iterations to converge when the number of parameters increases (due to the max norm adopted here as stopping criterion). Nevertheless, we remark that no thresholding on the matrix $\boldsymbol{S}$ has been applied in both cases; thus, in the same model even tasks with small similarities can be included, which in principle might introduce noise in the learning and inference processes. Consequently, the advantage of jointly learning a larger number of similar tasks can be compensated by this potential noise; investigating other task grouping and similarity thresholding strategies could thereby give rise to further insights about model, which for lack of room we destine to future study.

Regarding the impact of parameter $\beta$, regulating the effect of dynamics regularization, a strong decay in performance is obtained when no regularization is applied ($\beta=0$): this confirms the tendency of the network trajectory to be attracted in some limit cases by trivial fixed points, already observed in the singletask Hopfield model [13]. In this experiment, the contribution of regularization is still more dominating, since it allows to double the AUPR performance.

Less impact on the performance apparently has the parameter $\alpha$, regulating the multitask energy term (Eq. (3)). Indeed, the performance reduces just of $0.008$ when $\alpha=0$; however, this behaviour should be further studied, because it can be strictly related to the noise we introduced by grouping tasks without filtering out connections between less similar task. Thus, further experiments with different organisms would help this analysis and potentially reveal novel and more clear trends. It is also important noting that by setting $\alpha=0$, the overall multitask contribution is not cancelled: the learning procedure, by maximizing criterion (10), still learns tasks jointly, even when the multitask contribution in formula (9) is removed. For instance, choosing $\sigma$ equal to the minimum function would mean learning individual task parameters in order to maximize the minimum performance ($\min_{k}F_{k}$) across tasks, even when $\alpha=0$.

Finally, the function $\sigma$ itself seems having a marked impact on the model. When using the mean function ($\sigma_{2}$) the AUPR decreases of around $25\%$ with respect to the AUPR obtained using the harmonic mean ($\sigma_{1}$). To some extent such a result is expected, since the harmonic mean tends to penalize more the outliers towards [math], thus fostering the learning procedure to estimate the parameters in order not to penalize some tasks in favors of the remaining ones, which instead can happen when using the mean function.

This preliminary analysis about the model suggested to adopt the configuration “Card, $\sigma_{1}$” in the comparison with the state-of-the-art methodologies, which is described in the next section.

3.4 Model performance

We compared our method with several state-of-the-art graph-based algorithms, ranging from singletask Hopfield networks and other multitask methodologies, to some methods specifically designed to predicting protein functions: RW, random walk [30], the classical $t$-step random walk algorithm, predicting a score corresponding to the probability that a $t$-step random walk in $G$, starting from positive nodes, ends in the node to be predicted; RWR, random walk with restart, since in RW after many steps the walker may forget the prior information coded in the initial probability vector (0 for negative nodes $1/|L_{k,+}|$ for positive nodes), RWR allows the walker to move another random walk step with probability $1-\theta$, or to restart from its initial condition with probability $\theta$; GBA, guilt-by-association [34], a method based on the assumption that interacting proteins are more likely to share similar functions; LP, label propagation [43], a popular semi-supervised learning algorithm which propagates labels to unlabeled nodes through an iterative process based on Gaussian random fields over a continuous state space; MTLP, MTLP-inv [14], two recent multitask extensions of LP, exploiting task dissimilarities (MTLP) and similarities (MTLP-inv); MS-kNN, Multi-Source k-Nearest Neighbors [29], a method based on the $k$-Nearest Neighbours ($k$NN) algorithm [1], among the top-ranked methods in the recent CAFA2 international challenge for AFP [25]; RANKS [36], a recent graph-based method proposed to rank proteins, adopting a suitable kernel matrix to extend the notion of node similarity also to non neighboring nodes; COSNet. We used the same approach in [18] to compute a node ranking, necessary to calculating both AUC and AUPR.

In Table 2 we show the obtained results. Our method achieves the highest AUPR in all the experiments, with statistically significant difference over the second top method (RWR) in $2$ out of $3$ experiments (Wilcoxon signed rank test, $\alpha=0.05$). The performance improvement compared with COSNet is noticeable, showing the remarkable contribution supplied by our multitask extension. Interestingly, MTLP and MTLP-inv increase the AUPR results of LP not so remarkably as HoMTask: this means that the further information regarding task similarities should be appropriately exploited in order to achieve relevant gains. RANKS is the third method in all experiments, followed by MTLP(-inv), while MS-$k$NN is surprisingly the last method. Our method achieves good results also in terms of AUC (which however is less informative in this context), being close to top performing methods (RWR on CC and MF, and MTLP-inv on BP terms).

Conclusions

We have proposed the first multitask Hopfield Network for classification purposes, HoMTask, capable to simultaneously learn multiple tasks and to cope with the label imbalance. In our validation experiments, it significantly outperformed singletask HNs, and favorably compared with state-of-the-art single and multitask graph-based methodologies. Future investigations might reveal novel insights about the model, in particular regarding the choice of the task relatedness matrix, the task grouping strategy, the multitask criterion to be optimized during the learning phase, and the optimization procedure itself.