Multitask diffusion adaptation over networks with common latent representations

TL;DR

This paper introduces a novel distributed online multitask learning framework where multiple tasks share a common latent feature representation, enabling efficient collaboration among agents in streaming data environments.

Contribution

It proposes a new multitask learning model based on shared latent representations and develops distributed algorithms with a unified mean-square-error analysis.

Findings

Algorithms demonstrate effective multitask learning with shared features.

Theoretical analysis confirms stability and convergence properties.

Simulations validate the approach's potential in practical applications.

Abstract

Online learning with streaming data in a distributed and collaborative manner can be useful in a wide range of applications. This topic has been receiving considerable attention in recent years with emphasis on both single-task and multitask scenarios. In single-task adaptation, agents cooperate to track an objective of common interest, while in multitask adaptation agents track multiple objectives simultaneously. Regularization is one useful technique to promote and exploit similarity among tasks in the latter scenario. This work examines an alternative way to model relations among tasks by assuming that they all share a common latent feature representation. As a result, a new multitask learning formulation is presented and algorithms are developed for its solution in a distributed online manner. We present a unified framework to analyze the mean-square-error performance of the adaptive strategies, and conduct simulations to illustrate the theoretical findings and potential applications.

Figures

Equations

$$d_k(n)={\mathbf{x}}_{k,n}{\mathbf{w}}_k^o+z_k(n)$$

$$J_k(\mathbf{w})=\mathbb{E}\{|d_k(n)-{\mathbf{x}}_{k,n}\mathbf{w}{|}^2\}$$

$${\mathbf{w}}_k^o={\mathbf{w}}^o$$

$$J^{\text{glob}}(\mathbf{w})=\sum _{k=1}^N{J}_k(\mathbf{w})$$

$${\mathbf{w}}_k^o\sim {\mathbf{w}}_{\ell }^o\text{ if }\ell \in {\mathcal{N}}_k$$

$${\mathbf{w}}_k^o=\mathbf{\Theta }{\mathbf{u}}^o+{\mathbf{ϵ}}_k^o$$

$$J^{\text{glob}}\,\big{(}\boldsymbol{u},\{\boldsymbol{\epsilon}_{k}\}_{k=1}^{N}\big{)}=\sum_{k=1}^{N}\mathbb{E}\big{\{}|d_{k}(n)-\boldsymbol{x}_{k,n}(\boldsymbol{\Theta}\boldsymbol{u}+\boldsymbol{\epsilon}_{k})|^{2}\big{\}}$$

$$\begin{split}\min_{\boldsymbol{u},\{\boldsymbol{\epsilon}_{k}\}_{k=1}^{N}}&J^{\text{glob}}\big{(}\boldsymbol{u},\{\boldsymbol{\epsilon}_{k}\}_{k=1}^{N}\big{)}\\ &\text{ subject to }\boldsymbol{\epsilon}_{k}\in\text{span}(\boldsymbol{\Theta}_{\perp}),\quad\forall k=1,\dots,N\end{split}$$

$${\mathbf{ϵ}}_k={\mathbf{\Theta }}_{\perp }{\mathbf{\xi }}_k$$

$$\displaystyle J^{\text{glob}}\big{(}\boldsymbol{u},\{\boldsymbol{\xi}_{k}\}_{k=1}^{N}\big{)}$$

$$\displaystyle=\sum_{k=1}^{N}\mathbb{E}\big{\{}|d_{k}(n)-\boldsymbol{x}_{k,n}(\boldsymbol{\Theta}\boldsymbol{u}+\boldsymbol{\Theta}_{\perp}\boldsymbol{\xi}_{k})|^{2}\big{\}}$$

$$\displaystyle=\sum_{k=1}^{N}\mathbb{E}\{|d_{k}(n)|^{2}\}+\boldsymbol{u}^{*}\boldsymbol{\Theta}^{*}{\Big{(}}\sum_{k=1}^{N}\boldsymbol{R}_{x,k}{\Big{)}}\boldsymbol{\Theta}\boldsymbol{u}+\sum_{k=1}^{N}\boldsymbol{\xi}_{k}^{*}\boldsymbol{\Theta}_{\perp}^{*}\boldsymbol{R}_{x,k}\boldsymbol{\Theta}_{\perp}\boldsymbol{\xi}_{k}+2\,\text{Re}\Big{\{}\boldsymbol{u}^{*}\boldsymbol{\Theta}^{*}\sum_{k=1}^{N}\boldsymbol{R}_{x,k}\boldsymbol{\Theta}_{\perp}\boldsymbol{\xi}_{k}\Big{\}}$$

$$\displaystyle-2\,\text{Re}\Big{\{}\sum_{k=1}^{N}\boldsymbol{p}_{dx,k}^{*}\boldsymbol{\Theta}\boldsymbol{u}\Big{\}}-2\,\text{Re}\Big{\{}\sum_{k=1}^{N}\boldsymbol{p}_{dx,k}^{*}\boldsymbol{\Theta}_{\perp}\boldsymbol{\xi}_{k}\Big{\}}$$

$$J_u^{\text{glob}}(\mathbf{u})$$

$$=\sum _{k=1}^N{J}_{u,k}(\mathbf{u})$$

$$\begin{array}[]{lcl}\displaystyle\sum_{k=1}^{N}c_{\ell k}=1,&\text{ and }&c_{\ell k}=0\text{ if }k\notin{\mathcal{N}}_{\ell}\\ \end{array}$$

$$J_{u,k}^{\text{loc}}(\mathbf{u})=\sum _{\ell \in {\mathcal{N}}_k}{c}_{\ell k}{J}_{u,\ell }(\mathbf{u})$$

$$\begin{array}{rl}{J}_u^{\text{glob}}(\mathbf{u})& =J_{u,k}^{\text{loc}}(\mathbf{u})+\sum _{\ell \ne k}{J}_{\mathbf{u},\ell }^{\text{loc}}(\mathbf{u})\\ & =J_{u,k}^{\text{loc}}(\mathbf{u})+\sum _{\ell \ne k}\Vert \mathbf{u}-{\mathbf{u}}^o{\Vert }_{{\nabla }^2{J}_{u,\ell }^{\text{loc}}}^2\end{array}$$

$${\mathbf{\varphi }}_{k,n}$$

$${\mathbf{u}}_{k,n}$$

$$\begin{array}[]{lcl}\displaystyle\sum_{\ell=1}^{N}a_{\ell k}=1,&\text{ and }&a_{\ell k}=0\text{ if }\ell\notin{\mathcal{N}}_{k},\end{array}$$

$${\mathbf{\varphi }}_{k,n}$$

$${\mathbf{u}}_{k,n}$$

$$\begin{split}&J_{\xi}^{\text{glob}}\big{(}\{\boldsymbol{\xi}_{k}\}_{k=1}^{N}\big{)}\\ &=\sum_{k=1}^{N}\Big{(}\boldsymbol{\xi}_{k}^{*}\boldsymbol{\Theta}_{\perp}^{*}\boldsymbol{R}_{x,k}\boldsymbol{\Theta}_{\perp}\boldsymbol{\xi}_{k}+2\,\text{Re}\Big{\{}\boldsymbol{\xi}_{k}^{*}{\boldsymbol{\Theta}_{\perp}^{*}\boldsymbol{R}_{x,k}\boldsymbol{\Theta}\boldsymbol{u}^{o}}\Big{\}}-2\,\text{Re}\Big{\{}\boldsymbol{p}_{dx,k}^{*}\boldsymbol{\Theta}_{\perp}\boldsymbol{\xi}_{k}\Big{\}}\Big{)}+g^{\prime}_{k}(\boldsymbol{u}^{o})\\ &=\sum_{k=1}^{N}J_{\xi,k}(\boldsymbol{\xi}_{k})\end{split}$$

$$\begin{split}&\boldsymbol{\xi}_{k,n}=\boldsymbol{\xi}_{k,n-1}+\mu\boldsymbol{\Theta}_{\perp}^{*}\boldsymbol{x}_{k,n}^{*}\big{[}d_{k}(n)-\boldsymbol{x}_{k,n}(\boldsymbol{\Theta}\boldsymbol{u}_{k,n-1}{+}\,\boldsymbol{\Theta}_{\perp}\boldsymbol{\xi}_{k,n-1})\big{]}\end{split}$$

$${\mathbf{w}}_{k,n}=\mathbf{\Theta }{\mathbf{u}}_{k,n}+{\mathbf{\Theta }}_{\perp }{\mathbf{\xi }}_{k,n}$$

$${\mathbf{u}}_{k,n}$$

$${\mathbf{\xi }}_{k,n}$$

$${\mathbf{\psi }}_{k,n}$$

$$\displaystyle\stackrel{{\scriptstyle(b)}}{{=}}\boldsymbol{\Theta}\boldsymbol{u}_{k,n-1}+\boldsymbol{\Theta}_{\perp}\boldsymbol{\xi}_{k,n-1}+\mu\Big{[}(\boldsymbol{\Theta}\boldsymbol{\Theta}^{*}+\boldsymbol{\Theta}_{\perp}\boldsymbol{\Theta}_{\perp}^{*})\boldsymbol{x}_{k,n}^{*}[d_{k}(n)-\boldsymbol{x}_{k,n}(\boldsymbol{\Theta}\boldsymbol{u}_{k,n-1}+\boldsymbol{\Theta}_{\perp}\boldsymbol{\xi}_{k,n-1})]\Big{]}$$

$$\displaystyle=\boldsymbol{w}_{k,n-1}+\mu\boldsymbol{S}_{\boldsymbol{\Theta}}\boldsymbol{x}_{k,n}^{*}\big{[}d_{k}(n)-\boldsymbol{x}_{k,n}\boldsymbol{w}_{k,n-1}\big{]}$$

$${\mathbf{w}}_{k,n}$$

$$\stackrel{(c)}{=}\mathbf{\Theta }\sum _{\ell \in {\mathcal{N}}_k}{a}_{\ell k}({\mathbf{\Theta }}^{\ast }\mathbf{\Theta }{)}^{-1}{\mathbf{\Theta }}^{\ast }{\mathbf{\psi }}_{k,n}+{\mathbf{\Theta }}_{\perp }({\mathbf{\Theta }}_{\perp }^{\ast }{\mathbf{\Theta }}_{\perp }{)}^{-1}{\mathbf{\Theta }}_{\perp }^{\ast }{\mathbf{\psi }}_{k,n}$$

$$=\sum _{\ell \in {\mathcal{N}}_k}{a}_{\ell k}{\mathbf{P}}_{\mathbf{\Theta }}{\mathbf{\psi }}_{\ell ,n}+{\mathbf{P}}_{{\mathbf{\Theta }}_{\perp }}{\mathbf{\psi }}_{k,n}$$

$$\boldsymbol{\psi}_{k,n}=\boldsymbol{w}_{k,n-1}+\mu\,\boldsymbol{x}_{k,n}^{*}\big{[}d_{k}(n)-\boldsymbol{x}_{k,n}\boldsymbol{w}_{k,n-1}\big{]}$$

$${\mathbf{\psi }}_{k,n}$$

Full text

I Introduction

Multi-agent networks usually consist of a large number of interconnected agents or nodes. Interconnections between the agents allow them to share information and collaborate in order to solve complex tasks collectively. Examples abound in the realm of social, economic and biological networks. Distributed algorithms over such networks offer a valuable alternative to centralized solutions with useful properties such as scalability, robustness, and decentralization. When endowed with adaptation abilities, these algorithms enable agents to continuously learn and adapt in an online manner to concept drifts in their data streams [Sayed2014IEEE, Sayed2014adaptation]. Broadly, distributed strategies for online parameter estimation can be applied to single-task or multi-task scenarios. In the first case, agents cooperate with each other to estimate a single parameter vector of interest, such as tracking a common target. Reaching consensus among the agents is critical for successful inference in these problems. In the multitask case, the agents cooperate to estimate multiple parameter vectors simultaneously, such as tracking a collection of targets moving in formation [Chen2014multitask].

Extensive studies have been conducted on adaptive distributed strategies for single-task problems. Existing techniques include incremental [Bertsekas1997, Rabbat2005, Blatt2007, Lopes2007incr], consensus [Nedic2009, Kar2009, Srivastava2011], and diffusion strategies [Chen2014MLSP, Sayed2014IEEE, Sayed2013diff, Sayed2013intr, Lopes2008diff, Cattivelli2010diff, ChenUCLA2012, ChenUCLA2013]. Incremental techniques require determining a cyclic path that runs across all nodes, which is generally a challenging (NP-hard) task to perform. Besides, feature makes the incremental strategies sensitive to link failures and problematic for adaptation. Consensus techniques aim to reach an agreement among nodes on the estimate of interest via local information exchanges, but they have been shown [Sayed2014IEEE, Sayed2014adaptation] to suffer from instability problems when used in the context of adaptive networks due to an inherent asymmetry in the update equations. Diffusion techniques, on the other hand, have been shown to have superior stability and performance ranges [Tu2012] than consensus-based implementations. For these reasons, we shall focus on diffusion-type implementations in this paper.

Besides single-task scenarios, there are also applications where it is desirable to estimate multiple parameter vectors at the same time, rather than promote consensus among all agents [Chen2015diffusion]. For example, geosensor networks that monitor dynamic spatial fields, such as temperature or windspeed variations in geographic environments, require node-specific estimation problems that are able to take advantage of the spatial correlation between the measurements of neighboring nodes [Abdolee2014, Hirata2008]. A second example is the problem of collaborative target tracking where agents track several objects simultaneously [Chen2014multitask, Chen2015diffusion]. Motivated by these applications, there have been several variations of distributed strategies to deal with multitask scenarios as well. Existing strategies mostly depend on how the tasks relate to each other and on exploiting some prior information. In a first scenario, nodes are grouped into clusters, and each cluster of nodes is interested in estimating its own parameter vector. Although clusters may generally have distinct though related estimation tasks to perform, the nodes may still be able to capitalize on inductive transfer between clusters to improve their estimation accuracy. Multitask diffusion strategies were developed to perform estimation under these conditions [Chen2014multitask, nassif2015multitask]. One useful way to do so is to employ regularization. A couple of other useful works have also addressed variations of this scenario where the only available information is that clusters may exist in the network but nodes do not know which other nodes share the same estimation task [Zhao2015, Chen2015Eusipco, nassif2016diffusion]. In [Monajemi2016], the authors use multitask diffusion adaptation with a node clustering strategy to identify a model between the gait information and electroencephalographic signals. In [Wang2016], the authors consider the framework in [Chen2014multitask] to devise a distributed strategy that allows each node in the network to locally adapt inter-cluster cooperation weights. The authors in [Roula2016] promote cooperation between clusters with $\ell_{1}$-norm co-regularizers. They derive a closed-form expression of the proximal operator, and introduce a strategy that also allows each node to automatically set its inter-cluster cooperation weights. The works in [Monajemi2015, Khawatmi2015] propose alternative node clustering strategies. In a second scenario, it is assumed that there are parameters of global interest to all nodes in the network, a collection of parameters of common interest within sub-groups of nodes, and a set of parameters of local interest at each node. A diffusion strategy was developed to perform estimation under these conditions [Plata2015TSP, Plata2016ICASSP]. Likewise, in the works [Bertrand2010P1, Bertrand2011, Bogdanovic2014TSP], distributed algorithms are derived to estimate node-specific parameter vectors that lie in a common latent signal subspace. In another work [Chen2016IcasspGroup], the diffusion LMS algorithm is extended to deal with structured criteria built upon groups of variables, leading to a flexible framework that can encode various structures in the parameters. An unsupervised strategy to differentially promote or inhibit collaboration between nodes depending on their group is also introduced.

Alternatively, in recent years, there has been an increasing interest in modeling relations between tasks by assuming that all tasks share a common feature representation in a latent subspace [Baxter2000, Ando2005, Zhou2011]. The authors in [Ando2005] proposed a non-convex method based on Alternating Structure Optimization (ASO) for identifying the task structure. A convex relaxation of this approach was developed in [Chen2009mtl]. In [Zhou2011], the authors showed the equivalence between ASO, clustered multitask learning [Jacob2009, Obozinski2010] and their convex relaxations. The efficiency of such task relationships has been demonstrated in these works for clustering and classification problems. In our preliminary work [Chen2014MLSP], we introduced this framework within the context of distributed online adaptation over networks. Useful applications can be envisaged. First, consider the case where the common subspace is spanned by certain selected columns of the identity matrix. This means that a subset of the entries of the parameter vector to be estimated are common to all nodes while no further restriction is imposed on the other entries. Another example concerns beamforming for antenna arrays with a generalized side-lobe canceller (GSC). The latent subspace corresponds to the space where interfering signals reside [Griffiths1982]. A third example deals with cooperative spectrum sensing in cognitive radios, where the common latent subspace characterizes common interferers [Plata2015TSP].

Drawing on these motivations, this paper deals with distributed learning and adaptation over multitask networks with common latent representation subspaces. Algorithms are designed accordingly, and their performance analyzed. The contributions of this work include the following main aspects:

• •

We formulate a new multitask estimation problem, which assumes that all tasks share a common latent subspace representation in addition to node-specific contributions. Additional constraints can be incorporated if needed. This work contrasts with earlier works [Chen2014multitask, Roula2016], where the inductive transfer between learning tasks is promoted by regularizers. It also differs from [Plata2015TSP], which considers direct models by stacking local and global variables in an augmented parameter vector. Moreover, the work [Ando2005] uses a similar inductive transfer model but the common latent subspace is unknown and embedded into a joint estimation process. Our work is the first one to introduce an online estimation algorithm over networks. Estimating the common latent subspace of interest within this context is a challenging perspective.

• •

We explain how this formulation can be tailored to fit individual application contexts by considering additional model constraints. We illustrate this fact by considering two convex optimization problems and the associated distributed online algorithms. The first algorithm is a generalization in some sense of the diffusion LMS algorithm, which can be retrieved by defining the low-dimensional common latent subspace as the whole parameter space. The second algorithm uses $\ell_{2}$-norm regularization to account for the multitask nature of the problem. This opens the way to other regularization schemes depending on the application at hand.

• •

We present a unified framework for analyzing the performance of these algorithms. This framework also allows to address the performance analysis of the multitask algorithms in [Zhao2012, Chen2014multitask, nassif2014performance, Chen2015diffusion] in a generic manner, though these analyses were performed independently of each other in these works.

The rest of the paper is organized as follows. Section II introduces the multitask estimation problem considered in this paper. Then, two distributed learning strategies are derived in Section III by imposing different constraints on common and node-specific representation subspaces. Section IV provides a general framework for analyzing distributed algorithms of this form. In Section V, experiments are conducted to illustrate the characteristics of these algorithms. Section VI concludes the paper and connects our work with several other learning strategies.

Notation. Normal font $x$ denotes scalars. Boldface small letters $\boldsymbol{x}$ denote vectors. All vectors are column vectors. Boldface capital letters ${\boldsymbol{X}}$ denote matrices. The asterisk $(\cdot)^{*}$ denotes complex conjugation for scalars and complex-conjugate transposition for matrices. The superscript $(\cdot)^{\top}$ represents transpose of a matrix or a vector, and $\|\!\cdot\!\|$ is the $\ell_{2}$-norm of its matrix or vector argument. $\text{Re}\{\cdot\}$ and $\text{Im}\{\cdot\}$ denote the real and imaginary parts of their complex argument, respectively. Matrix trace is denoted by trace$(\cdot)$. The operator $\text{col}\{\cdot\}$ stacks its vector arguments on the top of each other to generate a connected vector. The operator $\text{diag}\{\cdot\}$ formulates a (block) diagonal matrix with its arguments. Identity matrix of size $N\times N$ is denoted by $\boldsymbol{I}_{N}$. Kronecker product is denoted by $\otimes$, and expectation is denoted by $\mathbb{E}\{\cdot\}$. We denote by ${\mathcal{N}}_{k}$ the set of node indices in the neighborhood of node $k$, including $k$ itself, and $|{\mathcal{N}}_{k}|$ its set cardinality.

II Matched subspace estimation over multitask networks

II-A Multitask estimation problems over networks

Consider a connected network composed of $N$ nodes. The problem is to estimate an $L\times 1$ unknown vector $\boldsymbol{w}_{k}^{o}$ at each node $k$ from collected measurements. At each time $n$, node $k$ has access to local streaming measurements $\{d_{k}(n),\boldsymbol{x}_{k,n}\}$, where $d_{k}(n)$ is a scalar zero-mean reference signal, and $\boldsymbol{x}_{k,n}$ is a $1\times L$ zero-mean row regression vector with covariance matrix $\boldsymbol{R}_{x,k}=\mathbb{E}\{\boldsymbol{x}_{k,n}^{*}\boldsymbol{x}_{k,n}\}>0$. The data at agent $k$ and time $n$ are assumed to be related via the linear model:

[TABLE]

where $\boldsymbol{w}_{k}^{o}$ is an unknown complex parameter vector, and $z_{k}(n)$ is a zero-mean i.i.d. noise with variance $\sigma_{z,k}^{2}=\mathbb{E}\{|z_{k}(n)|^{2}\}$. The noise signal $z_{k}(n)$ is assumed to be independent of any other signal. Let $J_{k}(\boldsymbol{w})$ be a differentiable convex cost function at agent $k$. In this paper, we shall consider the mean-square-error criterion:

[TABLE]

It is clear from (1) that each $J_{k}(\boldsymbol{w})$ is minimized at $\boldsymbol{w}_{k}^{o}$. We refer to each parameter $\boldsymbol{w}_{k}^{o}$ to estimate (or model in a more general sense) as a task. Depending on whether the minima of all $J_{k}(\boldsymbol{w})$ are achieved at the same $\boldsymbol{w}_{k}^{o}$ or not, the distributed learning problem can be single-task or multitask oriented [Chen2014multitask].

With single-task networks, all agents aim at estimating the same parameter vector $\boldsymbol{w}^{o}$ shared by the entire network, that is,

[TABLE]

for all $k\in\{1,...,N\}$. Several popular collaborative strategies, such as diffusion LMS [Chen2014MLSP, Sayed2014IEEE, Lopes2008diff, Sayed2013intr], were derived to address this problem by seeking the minimizer of the following aggregate cost function:

[TABLE]

in a distributed manner. Since the individual costs (2) admit the same solution, $\boldsymbol{w}^{o}$ is also the solution of (4). It has been shown that using proper cooperative strategies to solve (4) can improve the estimation performance [Sayed2014IEEE, Sayed2014adaptation].

With multitask networks, each agent aims at determining a local parameter vector $\boldsymbol{w}_{k}^{o}$. It is assumed that some similarities or relations exist among the parameter vectors of neighboring agents so that cooperation can still be meaningful, namely,

[TABLE]

where the symbol $\sim$ refers to a similarity relationship in some sense, which can be exploited to enhance performance. Depending on the problem characteristics, this property can be promoted in several ways, e.g., by introducing some regularization term, or by assuming a common latent structure. Networks may also be structured into clusters where agents within each cluster estimate the same parameter vector [Chen2014multitask, Zhao2012].

II-B Node-specific subspace constraints

Although agents aim to estimate distinct minimizers $\boldsymbol{w}_{k}^{o}$, exploiting relationships between solutions can make cooperation among agents beneficial. Regularization is one popular technique for introducing prior information about the solution. It can improve estimation accuracy though it may introduce bias [Chen2015diffusion, Chen2014multitask, Chen2013performance]. In this paper, we explore an alternative strategy that assumes that the hypothesis spaces partially overlap. Specifically, we assume that each $\boldsymbol{w}_{k}^{o}$ can be expressed in the form:

[TABLE]

where $\boldsymbol{\Theta}\boldsymbol{u}^{o}$ is common to all nodes with $\boldsymbol{\Theta}$ denoting an $L\times M$ matrix with known entries and ${\boldsymbol{u}}^{o}$ an unknown $M\times 1$ parameter vector (common to all nodes), and where $\boldsymbol{\epsilon}_{k}^{o}$ is an unknown node-specific component. We assume that matrix $\boldsymbol{\Theta}=[\boldsymbol{\theta}_{1},\dots,\boldsymbol{\theta}_{M}]$ is full-rank with $M\leq L$. Overcomplete sets of column vectors $\{\boldsymbol{\theta}_{1},\dots,\boldsymbol{\theta}_{M}\}$ may be advantageous in some scenarios but this usually requires to impose further constraints such as sparsity over $\boldsymbol{u}^{o}$. We shall not discuss this case further in order to focus on the main points of the presentation. Model (6) means that all tasks share the same parameter vector $\boldsymbol{\Theta}\boldsymbol{u}^{o}$, which lies in the subspace spanned by columns of $\boldsymbol{\Theta}$. This subspace representation can be useful in several applications. For instance, consider the case where $\boldsymbol{\Theta}$ is composed of selected columns of the identity matrix $\boldsymbol{I}_{L}$. This means that a subset of the entries of $\boldsymbol{w}^{o}_{k}$ are common to all agents while no further assumptions are imposed on the other entries. This situation is a natural generalization of the single-task scenario. Another example concerns beamforming problems with a generalized sidelobe canceller (GSC), where $\boldsymbol{\Theta}$ acts as a blocking matrix to cancel signal components that lie in the constraint space [Griffiths1982]. In machine learning, formulation (6) is referred to as the alternating structure optimization (ASO) problem [Ando2005, Zhou2011]. The subspace $\boldsymbol{\Theta}$ is, however, learnt simultaneously via a non-convex optimization procedure. In what follows, we shall assume that $\boldsymbol{\Theta}$ is known by each agent.

Before proceeding further, we clarify the difference between model (6) addressed here and in our preliminary work [Chen2014MLSP], and the model studied in [Bogdanovic2014TSP, Plata2013, Plata2016ICASSP, Bogdanovic2014, Plata2015TSP]. In these last works, the authors consider particular information access models where global and local components are assumed to be related to distinct regressors. The centralized problem can then be formulated by stacking the global and local regressors, and by considering a parameter vector augmented accordingly. In our work, motivated by applications of the latent space model in batch-mode learning, we address the problem where the parameter vectors to be estimated lie in global and local latent subspaces. We do not need to distinguish explicitly between global and local regressors. Instead, as shown in the sequel, some extra conditions are needed so that model (6) is identifiable. Among other possibilities, we shall investigate two strategies where constraints on $\boldsymbol{\Theta}$ and $\boldsymbol{\epsilon}_{k}^{o}$ are imposed.

Replacing (6) into (2), the global cost function is expressed as a function of a common parameter $\boldsymbol{u}$ and node-specific perturbations $\{\boldsymbol{\epsilon}_{k}\}_{k=1}^{N}$:

[TABLE]

We expect the estimation of $\boldsymbol{w}_{k}^{o}$ by each agent to benefit from the cooperative estimation of $\boldsymbol{u}$. Problem (7) is still insufficient for estimating the tasks $\{\boldsymbol{w}_{k}^{o}\}$. This is because the decomposition $\boldsymbol{w}_{k}=\boldsymbol{\Theta}\,\boldsymbol{u}+\boldsymbol{\epsilon}_{k}$ is not unique. Indeed, given any optimum solution $\{\bar{\boldsymbol{u}},\bar{\boldsymbol{\epsilon}}_{k}\}$, and any ${\boldsymbol{s}}=\boldsymbol{\Theta}\boldsymbol{x}$, we can generate another optimum solution by considering the shift $\{\bar{\boldsymbol{u}}-{\boldsymbol{x}},\bar{\boldsymbol{\epsilon}}_{k}+{\boldsymbol{s}}\}$. This ambiguity prevents us from deriving collaboration strategies based on $\boldsymbol{u}$. From the point of view of convex analysis, the Hessian matrix of (7) is rank deficient and no unique solution exists.

III Problem formulations and solution algorithms

Problem (7) can be modified to make it well-determined and more meaningful. In this section, among other possibilities, we investigate two strategies that consist of imposing further constraints and derive the corresponding distributed algorithms. These two formulations guarantee the uniqueness of the solution and have clear interpretations.

III-A Node-specific subspace constraints

We restrict the node-specific components $\{\boldsymbol{\epsilon}_{k}\}_{k=1}^{N}$ to lie in the complementary subspace to $\text{span}(\boldsymbol{\Theta})$. The problem can be formulated as:

[TABLE]

where the $L-M$ columns of matrix $\boldsymbol{\Theta}_{\perp}$ span the complementary subspace to $\text{span}(\boldsymbol{\Theta})$, that is, $\boldsymbol{\Theta}^{*}\,\boldsymbol{\Theta}_{\perp}={\boldsymbol{0}}$. We write:

[TABLE]

where $\boldsymbol{\xi}_{k}$ is a column vector of size $(L-M)$. Now, replacing (9) into (8), the optimization problem becomes unconstrained and the objective function is given by:

[TABLE]

where $\boldsymbol{R}_{x,k}=\mathbb{E}\{\boldsymbol{x}_{k,n}^{*}\boldsymbol{x}_{k,n}^{\phantom{*}}\}$ is the covariance matrix of $\boldsymbol{x}_{k,n}$, and $\boldsymbol{p}_{dx,k}=\mathbb{E}\{d_{k}(n)\boldsymbol{x}^{*}_{k,n}\}$ is the covariance vector between the input data $\boldsymbol{x}_{k,n}$ and the reference output data $d_{k}(n)$.

Lemma 1

Problem (8) has a unique solution with respect to $\boldsymbol{u}$ and $\{\boldsymbol{\epsilon}_{k}\}_{k=1}^{N}$ if the perturbations $\{\boldsymbol{\epsilon}_{k}\}_{k=1}^{N}$ lie in a subspace orthogonal to $\text{span}(\boldsymbol{\Theta})$.

Proof of Lemma 1 is provided in Appendix LABEL:app:proof-lemma1. We shall now derive a distributed algorithm to seek the minimizer of (8). Focusing on the terms that depend on $\boldsymbol{u}$ in (III-A), and setting parameters $\boldsymbol{\xi}_{k}$ to their optimum values $\boldsymbol{\xi}_{k}^{o}$, we consider first the global cost function over the variable $\boldsymbol{u}$:

[TABLE]

where $g_{k}(\boldsymbol{\xi}_{k}^{o})$ collects all the terms depending only on $\boldsymbol{\xi}_{k}^{o}$ in (III-A). The term $\sum_{k=1}^{N}\mathbb{E}\{|d_{k}(n)|^{2}\}$ is discarded because it is constant with respect to the arguments $\boldsymbol{u}$ and $\{\boldsymbol{\xi}_{k}\}_{k=1}^{N}$. Since $J_{u}^{\text{glob}}(\boldsymbol{u})$ has a unique minimizer for all nodes over the network, we can use a single-task adapt-then-combine (ATC) diffusion strategy to estimate $\boldsymbol{u}^{o}$ [Sayed2013intr, Cattivelli2010diff]. We introduce a right-stochastic matrix $\boldsymbol{C}$ with nonnegative entries $c_{\ell k}$ such that:

[TABLE]

With each node $k$, we associate the local cost over the variable $\boldsymbol{u}$:

[TABLE]

Observe that $\sum_{k=1}^{N}J_{u,k}^{\text{loc}}(\boldsymbol{u})=J_{u}^{\text{glob}}(\boldsymbol{u})$ because matrix $\boldsymbol{C}$ is right-stochastic. Since $J_{u}^{\text{glob}}(\boldsymbol{u})$ is quadratic with respect to $\boldsymbol{u}$, it can be expressed at each node $k$ as follows:

[TABLE]

where $\nabla^{2}J_{u,\ell}^{\text{loc}}$ denotes the Hessian matrix of $J_{u,\ell}^{\text{loc}}(\boldsymbol{u})$ with respect to $\boldsymbol{u}$, and $\|\boldsymbol{u}\|^{2}_{\boldsymbol{\Sigma}}$ is the squared norm of $\boldsymbol{u}$ weighted by any positive semi-definite matrix $\boldsymbol{\Sigma}$, i.e., $\|\boldsymbol{u}\|^{2}_{\boldsymbol{\Sigma}}=\boldsymbol{u}^{*}\boldsymbol{\Sigma}\boldsymbol{u}$. Following an argument based on the Rayleigh-Ritz characterization of eigenvalues [Sayed2013intr, Sec. 3.1], we approximate ${\nabla^{2}J_{\boldsymbol{u},\ell}^{\text{loc}}}$ by a multiple of the identity matrix, so that $\|\boldsymbol{u}-\boldsymbol{u}^{o}\|^{2}_{\nabla^{2}J_{\boldsymbol{u},\ell}^{\text{loc}}}\approx b_{\ell k}\|\boldsymbol{u}-\boldsymbol{u}^{o}\|^{2}$.

Minimizing (14) in two successive steps yields:

[TABLE]

where $\mu$ is a positive step size. Its choice to ensure stability of the algorithm will be elaborated on later in Sec. IV. Now, note the following. First, iteration (16) requires knowledge of $\boldsymbol{u}^{o}$, which is not available. Each node $\ell$ has a readily available, however, an approximation for $\boldsymbol{u}^{o}$, which is ${\boldsymbol{\phi}}_{k,n}$. Therefore, we replace $\boldsymbol{u}^{o}$ by ${\boldsymbol{\phi}}_{k,n}$ in (16). Second, ${\boldsymbol{\phi}}_{k,n}$ at node $k$ is generally a better estimate for $\boldsymbol{u}^{o}$ than $\boldsymbol{u}_{k,n-1}$ since it is obtained by incorporating information from the neighbors through (15). Therefore, we replace $\boldsymbol{u}_{k,n-1}$ by ${\boldsymbol{\phi}}_{k,n}$ in (16). Then, absorbing coefficients $b_{\ell k}$ into another set of nonnegative coefficients that satisfies:

[TABLE]

which means that matrix $\boldsymbol{A}$ with entries $a_{\ell k}$ is left-stochastic, using an instantaneous approximation of the gradient, and limiting the summation in (16) to the neighbors of node $\ell$ (see [Sayed2013intr, Cattivelli2010diff] for more details on a similar derivation in the context of single-task diffusion strategies), we can update $\boldsymbol{u}_{k,n}$ as follows:

[TABLE]

where $\boldsymbol{\xi}_{k,n-1}$ is an estimate for the unknown minimizer $\boldsymbol{\xi}_{k}^{o}$, to be evaluated as explained further ahead in (21).

Focusing on the terms that depend on $\{\boldsymbol{\xi}_{k}\}_{k=1}^{N}$ in (III-A), and setting parameter $\boldsymbol{u}$ to its optimum value $\boldsymbol{u}^{o}$, we consider the global cost function over the variables $\boldsymbol{\xi}_{k}$:

[TABLE]

where $g^{\prime}_{k}(\boldsymbol{\xi}_{k}^{o})$ collects all the terms depending only on $\boldsymbol{u}^{o}$ in (III-A). Now since the parameters $\boldsymbol{\xi}_{k}$ are node-specific, if no further constraints are imposed, they can be updated independently of each other via an LMS-type update:

[TABLE]

At each time instant $n$, node $k$ updates its parameters $\boldsymbol{u}_{k,n-1}$ and $\boldsymbol{\xi}_{k,n-1}$ using (18)–(19) and (21), respectively. The local estimate $\boldsymbol{w}_{k,n}$ is then given by:

[TABLE]

It is interesting to note that we can rewrite the algorithm without using the auxiliary variables $\boldsymbol{u}_{k{,n}}$ and $\{\boldsymbol{\xi}_{k{,n}}\}_{k=1}^{N}$, by substituting the relations:

[TABLE]

into (18)–(19) and (21), respectively. Selecting $\boldsymbol{C}=\boldsymbol{I}_{N}$ to avoid exchanging raw data and node-specific components, we can implement the update of $\boldsymbol{w}_{k,n-1}$ to an intermediate value $\boldsymbol{\psi}_{k,n}$ as follows:

[TABLE]

with $\boldsymbol{S}_{\boldsymbol{\Theta}}=\boldsymbol{\Theta}\boldsymbol{\Theta}^{*}+\boldsymbol{\Theta}_{\perp}^{\phantom{\top}}\boldsymbol{\Theta}_{\perp}^{*}$. For step (a), we use (22) with the intermediate value $\boldsymbol{\phi}_{k,n}$ of $\boldsymbol{u}_{k,n}$ in (18) and $\boldsymbol{\xi}_{k,n}$. Step (b) follows from their adaptation steps (18) and (21). Now substituting (19) in (22) to aggregate the intermediate estimates of $\boldsymbol{u}_{k,n}$ from the neighbors of node $k$, we arrive at the combination step:

[TABLE]

where $\boldsymbol{P}_{\boldsymbol{\Theta}}=\boldsymbol{\Theta}(\boldsymbol{\Theta}^{*}\boldsymbol{\Theta})^{-1}\boldsymbol{\Theta}^{*}$ and $\boldsymbol{P}_{\boldsymbol{\Theta}_{\perp}}=\boldsymbol{I}_{{L}}-\boldsymbol{P}_{\boldsymbol{\Theta}}$ are the projection matrices over subspaces $\boldsymbol{\Theta}$ and $\boldsymbol{\Theta}_{\perp}$. For step (c), we use (23)–(24) with the intermediate estimate $\boldsymbol{\psi}_{k,n}$. Finally, we arrive at the ATC strategy summarized in Algorithm 1.

The first step in (28) is an adaptation step where node $k$ uses the data realizations $\{d_{k}(n),\boldsymbol{x}_{k,n}\}$ to update its existing estimate $\boldsymbol{w}_{k,n-1}$ to an intermediate value $\boldsymbol{\psi}_{k,n}$. All other nodes in the network are performing a similar step. The second step in (29) is an aggregation step. To update its intermediate estimate to $\boldsymbol{w}_{k,n}$, each node $k$ combines the existing estimates of its neighbors in the common latent subspace $\boldsymbol{\Theta}$ to build up a common representation, and refines it with a node-specific value in $\boldsymbol{\Theta}_{\perp}$. In the special case when $\boldsymbol{A}=\boldsymbol{I}_{N}$, so that no information exchange is performed, the ATC strategy reduces to a non-cooperative solution where each node $k$ runs its own individual descent algorithm.

Matrix $\boldsymbol{S}_{\boldsymbol{\Theta}}$ in the adaptation step (28) is positive-definite. It arises from the calculation of the gradient of (III-A) with respect to $\boldsymbol{u}$ and $\boldsymbol{\xi}_{k}$. The algorithm can be simplified by replacing $\boldsymbol{S}_{\boldsymbol{\Theta}}$ by $\boldsymbol{I}_{L}$ in (28) without compromising the convergence of the method (as analyzed further ahead in Section LABEL:sec:analyses). We then arrive at the recursion:

[TABLE]

Strictly speaking, observe that $\boldsymbol{S}_{\boldsymbol{\Theta}}=\boldsymbol{I}_{L}$ if, and only if, the columns of $\boldsymbol{\Theta}$ and $\boldsymbol{\Theta}_{\perp}$ form an orthonormal basis of ${\mathrm{I\!R}}^{L}$. Note that the adaptation step (27) is the LMS solution for minimizing the cost in (III-A) with respect to $\boldsymbol{w}_{k}$.

Before leaving this section, we would like to point out that the algorithm described in [Plata2015TSP], which addresses direct models by stacking global and local variables in an augmented parameter vector, may be used to solve problem (8), provided that an appropriate variable change is performed in order to make the latent variables $\boldsymbol{u}_{k,n}$ and $\boldsymbol{\xi}_{k,n}$ explicit in $\boldsymbol{w}_{k,n}$. The resulting algorithm has the same performance as Algorithm 1 defined by (28), (29), but, obviously, they do not have the same form since they do not operate in the same domain. This structural difference has a major consequence for Algorithm 1. As already explained, it can be further tuned by replacing the matrix ${\boldsymbol{S}}_{{\boldsymbol{\Theta}}}$ in (28) by any positive definite matrix while ensuring convergence of the method. This extra degree of freedom will be taken into account in the analysis of the algorithm, where the only condition on ${\boldsymbol{S}}_{{\boldsymbol{\Theta}}}$ is to be positive definite. We will also show that setting ${\boldsymbol{S}}_{{\boldsymbol{\Theta}}}$ to $\boldsymbol{I}_{L}$, besides simplifying Algorithm 1, can greatly improve its performance.

III-B Node-specific subspace constraints with norm-bounded projections

The second formulation we consider is to relax the constraint that node-specific components $\{\boldsymbol{\epsilon}_{k}\}_{k=1}^{N}$ must lie in $\text{span}(\boldsymbol{\Theta}_{\perp})$. We now assume that they are norm-bounded in some sense. The problem is formulated as follows:

[TABLE]

Since the objective function and the constraints are convex in $(\boldsymbol{u},\{\boldsymbol{\epsilon}_{k}\}_{k=1}^{N})$, the constrained problem (30) can be formulated as a regularized optimization problem that consists of minimizing a global cost of the form [boyd2004convex]:

[TABLE]

where $\eta_{1}$ and $\eta_{2}$ are positive regularization parameters that are related to the bounds $\nu_{1}$ and $\nu_{2}$.