On the Theory of Dynamic Graph Regression Problem

TL;DR

This paper introduces a framework for efficiently updating linear regression solutions on dynamic graphs using update-efficient matrix embeddings, enabling fast recalculations after graph modifications.

Contribution

It defines update-efficient matrix embeddings and demonstrates how they enable fast updates of regression solutions in dynamic graphs, including adjacency and Laplacian matrices.

Findings

Exact regression solutions can be updated in O(nm) time after graph updates.

The approach works for both adjacency and Laplacian matrix embeddings.

Experiments show high efficiency on synthetic and real-world graphs.

Abstract

Most of real-world graphs are dynamic, i.e., they change over time by a sequence of update operations. While the regression problem has been studied for static graphs and temporal graphs, it is not investigated for general dynamic graphs. In this paper, we study regression over dynamic graphs. First, we present the notion of update-efficient matrix embedding, that defines conditions sufficient for a matrix embedding to be effectively used for dynamic graph regression (under l2 norm). Then, we show that given a n*m update-efficient matrix embedding (e.g., the adjacency matrix) and after an update operation in the graph, the exact optimal solution of linear regression can be updated in O(nm) time for the revised graph. Moreover, we show that this also holds when the matrix embedding is the Laplacian matrix and the update operations are restricted to edge insertion/deletion. In the end, by conducting experiments over synthetic and real-world graphs, we show the high efficiency of updating the solution of graph regression.

Tables

Table 1. Table 1: Empirical evaluation over real-world graphs.

Dataset	node insertion		node deletion
	update time	scratch time	update time	scratch time
wiki-vote	193.76	1601.20	159.37	1451.28
lastfm-asia	38.08	261.41	31.06	256.83
soc-sign-bitcoinotc	23.94	102.99	21.31	101.14

Equations

$$argmi{n}_{\text{mathbold}x}||\text{mathbold}A\cdot \text{mathbold}x-\text{mathbold}b|{|}_p.$$

$$\text{mathbold}x=\text{mathbold}{A}^{†}\cdot \text{mathbold}b$$

$$argmi{n}_{\text{mathbold}}x||G\cdot \text{mathbold}x-\text{mathbold}b|{|}_2^2,$$

$$argmi{n}_{\text{mathbold}}x||\text{mathbold}M\cdot \text{mathbold}x-\text{mathbold}b|{|}_2^2.$$

$$\text{mathbold}{M}^{\prime }=\text{mathbold}M+\sum _{k=1}^K\left(\text{mathbold}{c}^k\cdot \text{mathbold}{d^k}^{\ast }\right).$$

$$\text{mathbold}{M}^{k+1}=\text{mathbold}{M}^k+\text{mathbold}{c}^k\cdot {\text{mathbold}{d}^k}^{\ast },$$

$$\text{mathbold}{M}^{\prime }=[\text{mathbold}M\text{mathbold}a],$$

$$\text{mathbold}{M}^{\prime }{}^{†}=\left[\begin{array}{c}\text{mathbold}{M}^{†}-\text{mathbold}d\cdot \text{mathbold}f\\ \text{mathbold}f\end{array}\right]$$

$$\text{mathbold}f=\{\begin{array}{ll}\text{mathbold}{c}^{†},& \text{if }\text{mathbold}c\ne \text{mathbold}\theta \\ (1+\text{mathbold}{d}^{\ast }\cdot \text{mathbold}d{)}^{-1}\text{mathbold}{d}^{\ast }\cdot \text{mathbold}{M}^{†},& \text{if }\text{mathbold}c=\text{mathbold}\theta \end{array}$$

$$\text{mathbold}{M}^{k+1}=\text{mathbold}{M}^k+\text{mathbold}{c}^k\cdot {\text{mathbold}{d}^k}^{\ast },$$

$$\text{mathbold}M=[\text{mathbold}{M}^{\prime }\text{mathbold}a],$$

$$argmi{n}_{\text{mathbold}x}||\text{mathbold}A\cdot \text{mathbold}x-\text{mathbold}b|{|}_1,$$

$$\text{centering}\text{mathbold}x=\text{mathbold}{M}^{†}\text{mathbold}b=\left[\begin{array}{c}-36497\\ 12368\\ 14436\\ -15438\\ 9706.3\\ 103000\\ 32985\end{array}\right].\text{@add@centering}$$

$$\text{centering}\text{mathbold}x=\left[\begin{array}{c}-37069\\ 12196\\ 15248\\ -15252\\ 9553.5\\ 102180\\ 32632\end{array}\right].\text{@add@centering}$$

Full text

Keywords

Dynamic graphs, linear regression, update-efficient matrix embeddings, update time

1 Introduction

Graphs are an important tool for modeling many complex systems such as the world wide web, social networks, road networks and citation networks. Most of real-world graphs are dynamic, i.e., they change over time by a sequence of graph update operations. A graph update operation can be either a node insertion, or a node deletion, or an edge insertion, or an edge deletion or an edge weight change.

A fundamental task in machine learning and data analysis is linear regression. In this problem, we receive $n$ data, where for each $i\in[1,n]$, the data consists of a row in a $n\times m$ matrix $\mathbold{A}$ and a single element in a $n\times 1$ vector $\mathbold{b}$. Then, the goal is to find a vector $\mathbold{x}$ such that $\mathbold{A}\cdot\mathbold{x}$ is the closest point to $\mathbold{b}$ in the column span of $\mathbold{A}$, under a proper distance measure, such as the $l_{2}$ norm (Euclidean distance or the least squares distance) or the $l_{1}$ norm (the least absolute deviation). More formally, we want to solve the following problem ($p\in[1,\infty)$):

[TABLE]

While this problem has been studied for (static) high dimensional data, static graphs [1] and temporal graphs [21], to the best of our knowledge it is not investigated for general dynamic graphs. We refer to the regression problem over dynamic graphs as dynamic graph regression. In the current paper, we study the theory of dynamic graph regression, over general graphs. The importance of this problem stems from a wide range of applications that generate dynamic graphs (hence, many data analysis algorithms have been developed for them [3, 23, 31]). Examples of these graphs include the world wide web, social networks and collaboration networks. We focus on the most common form of Equation 1 wherein $p$ is $2$, and is called least squares error (or $l_{2}$* norm*) linear regression.

The challenges of regression over dynamic graphs are twofold. The minor challenge is a proper adaptation of the standard setting of the regression problem to graphs. As we will discuss later, this can be done by using a matrix embedding for the graph. The major challenge arises due to the dynamic nature of the graph. This means when the structure of the graph changes by an update operation, the already found solution must be updated, to become valid for the revised graph. This is not trivial, as updating the solution must be done in a time considerably less than the time of computing it from scratch, for the revised graph.

In the current paper, we aim to address these challenges. First, we present the notion of update-efficient matrix embedding that defines conditions sufficient for a matrix embedding to be used for the dynamic graph regression problem (under $l_{2}$ norm). We show that some of the standard matrix embeddings, e.g., the (weighted) adjacency matrix, satisfy these conditions. Then, we show that given a $n\times m$ update-efficient matrix embedding, after an update operation in the graph, the exact optimal solution of the graph regression problem for the revised graph can be computed in $O(nm)$ time. In particular, using (weighted) adjacency matrix as the matrix embedding of the graph, it takes $O(n^{2})$ time to update the optimal solution, where $n$ is the number of nodes of the revised graph. Note that in this situation, computing the optimal solution for the revised graph from scratch will take $O(n^{3})$ time. We also show that similar results hold for the Laplacian matrix, if the update operations are limited to edge insertion and edge deletion. In the end, by performing experiments over synthetic and real-world graphs, we show the high empirical efficiency of the update method.

The rest of this paper is organized as follows. In Section 2, we provide an overview on related work. In Section 3, we introduce preliminaries and definitions used in the paper. In Section 4, we introduce update-efficient matrix embeddings. In Section 5, we study dynamic graph regression under least squares error. In Section 6, we present our experimental results. Finally, the paper is concluded in Section 7.

2 Related work

In recent years, several algorithms have been proposed for different learning problems over nodes of a static graph. Kleinberg and Tardos [26] studied the theory of the node classification problem and showed its connection to Markov random fields. With $k$ the number of node labels, they presented an $O(\log k\log\log k)$-approximation algorithm which has a polynomial time complexity. Kovac and Smith [1] developed a non-parametric regression model for nodes of a graph, where the distance between estimated values (vector $\mathbf{A}\mathbf{x}$) and measured values (vector $\mathbf{b}$) is computed using $l_{2}$ norm. Han et al. [22] proposed a representation learning method optimized for regression over nodes of a graph. Unlike these algorithms that deal with static graphs, in this paper we concentrate on efficiently updating the exact solution of least squares linear regression, over dynamic graphs.

There are a number of relevant problems in the literature. One of them is regression over temporal graphs, wherein node features (node contents) and measured values change over time, but the structure of the graph remains unchanged. This is in contrast with our studied problem which deals with changes in the structure of the graph. For example, while in dynamic graph regression we want to efficiently update the regression solution after adding a new edge to the graph, in temporal graph regression it is desired to update the solution when the values of the income feature (which is one of node features) of some nodes change. A common technique used to solve temporal graph regression is to utilize structural dependencies among measured values [34, 38, 40]. Han et al. [21] presented a different approach which is based on jointly learning embeddings for the measured values and node features.

The other relevant problem is online learning over nodes of a graph [25, 24]. Online learning is used where it is computationally infeasible to solve the learning task over the entire dataset. However in our studied dynamic graph regression problem, the task can be solved over the entire graph and the challenge is to effectively update the solution by structural changes in the graph. In one of known online graph learning algorithms, Herbster and Pontil [25] used a perceptron for online label prediction of nodes of a graph.

Another problem which has some connection to our studied problem is learning embeddings (representations) for nodes or subgraphs of a graph. In this problem, each node or subgraph is mapped to a vector in a low-dimensional vector space. In recent years, several algorithms have been proposed for it [19, 41, 32, 7], even though this task dates back to several decades ago. For example, Parsons and Pisanski [33] presented vector embeddings for nodes of a graph such that the inner product of the vector embeddings of any two nodes $i$ and $j$ is negative iff $i$ and $j$ are connected by an edge; and it is [math] otherwise. For an overview on learning embeddings for dynamic graphs, interested readers may refer to [30, 17, 16].

While the above mentioned methods learn with nodes or edges of a single graph, there also exist several algorithms in the literature that learn with a population of graphs [12, 37, 11, 28, 13, 5]. For example, Calissano et al. [5] studied building a regression model between a set of real values and a set of graphs.

Our studied setting is usually called dynamical setting  [20], wherein the structure of a single graph changes by means of a graph update operation, and the goal is to effectively update the solution of a problem for the new graph. The list of graph update operations that change the structure of the graph, is presented in Section 3. For a survey on different machine learning and data mining algorithms in the dynamical setting, interested readers may refer to [20].

3 Preliminaries

We use lowercase letters for scalars, uppercase letters for constants and graphs, bold lowercase letters for vectors and bold uppercase letters for matrices. We denote by $n$ the number of nodes of graph $G$. A dynamic graph is a graph that changes over time by a sequence of graph update operations [20]. A graph update operation is an operation that inserts an edge or a node into the graph; or deletes an edge or a node and its incident edges from the graph; or changes the weight of an edge. We assume that when a new node in inserted, some edges are also added between the new node and the existing nodes of the graph. We also assume that the new node obtains the largest node id of the graph.

The weighted adjacency matrix of $G$, denoted with $\mathbold{W}$, is a $n\times n$ matrix where $\mathbold{W}_{ij}$ contains the weight of the edge from node $i$ to node $j$, if $i$ has an edge to $j$ (otherwise, it is [math]). Given an undirected weighted graph $G$, its weighted Laplacian matrix $\mathbold{L}$ is a square matrix of size $n\times n$ defined as $\mathbold{L}=\mathbold{D}-\mathbold{W}$, where $\mathbold{D}$ is a $n\times n$ diagonal matrix with $\mathbold{D}_{ii}=\sum_{j=1}^{n}\mathbold{W}_{ij}$ and is called the (weighted) degree matrix. The Euclidean norm or $l_{2}$ norm of a vector $\mathbold{v}$ of size $n$, denoted with $||\mathbold{v}||_{2}$, is defined as $\sqrt{\mathbold{v}_{1}^{2}+\mathbold{v}_{2}^{2}+\cdots+\mathbold{v}_{n}^{2}}.$ Let $\mathbold{A}\in\mathbb{R}^{n\times m}$. The rank of $\mathbold{A}$ is the dimension of its column space (or its row space). By $\mathbold{A}^{*}$ we denote the transpose of $\mathbold{A}$ defined as an operator that switches the row and column indices of $\mathbold{A}$. The Singular Value Decomposition (SVD) of the $n\times m$ matrix $\mathbold{A}$ is defined as $\mathbold{U}\cdot\mathbold{\Sigma}\cdot\mathbold{V}^{*}$, where $\mathbold{U}$ is a $n\times m$ matrix with orthonormal columns, $\mathbold{\Sigma}$ is a $m\times m$ diagonal matrix with non-negative non-increasing entries on the diagonal, and $\mathbold{V}^{*}$ is a $m\times m$ matrix with orthonormal rows. The Moore-Penrose pseudoinverse matrix of $\mathbold{A}$, denoted by $\mathbold{A}^{\dagger}$, is the $m\times n$ matrix $\mathbold{V}\cdot\mathbold{\Sigma}^{{\dagger}}\cdot\mathbold{U}^{*}$, where $\mathbold{\Sigma}^{{\dagger}}$ is a $m\times m$ diagonal matrix defined as follows: $\mathbold{\Sigma}^{{\dagger}}_{i,i}=1/\mathbold{\Sigma}_{i,i}$, if $\mathbold{\Sigma}_{i,i}>0$, and [math] otherwise. It is well-known that the solution

[TABLE]

is an optimal solution (the closed form solution) for the linear regression problem under the $l_{2}$ norm, i.e., for the problem defined by Equation 1, with $p=2$ [35]. Time complexity of computing this solution is cubic in terms of $n$ and $m$ [35], which makes it slow and time consuming to compute it from scratch over large graphs.

4 Update-efficient matrix embeddings

By assuming that data are given in the form of a graph, we can extend the linear regression problem to the linear graph regression problem. In linear graph regression, we are given a graph $G$, with $n$ nodes, and a $n\times 1$ vector $\mathbold{b}$. Then, we want to solve the following optimization problem:

[TABLE]

where $G\cdot\mathbold{x}$ must satisfy the following conditions: i) $G\cdot\mathbold{x}$ is well-defined, and ii) $G\cdot\mathbold{x}$ must produce a $n\times 1$ column vector. A straightforward and common way to define $G\cdot\mathbold{x}$ is to replace $G$ by vector embeddings of nodes of $G$ (a matrix embedding111We note that this notion of embedding is different from the notion of embedding used in graph pattern mining [8, 9, 10]. of $G$), denoted with $\mathbold{M}$. As a result, for each node in the graph, we define a $1\times m$ row vector, and $\mathbold{x}$ is defined as a $m\times 1$ column vector. Hence, the linear graph regression problem is converted into finding the closest point to $\mathbold{b}$, in the column span of the matrix generated by the vector embeddings of nodes of $G$. In other words, we want to solve the following optimization problem

[TABLE]

As an example motivating linear graph regression problem, assume that we are given the graph of a social network, wherein each vertex is a person and the links represent the friendship relations. Moreover, a score is assigned to each vertex which determines e.g., its reputation. Now we want to find a function (with a least squares error) for the scores of the vertices, which is linear in terms of their structural properties. Therefore, we need to solve the linear graph regression problem for the given network. More precisely, first we need to compute a matrix embedding $\mathbold M$ of the nodes of the network, wherein each row $i$ is a representation of the structural properties of node $i$. Then we need to find a function for the scores which is linear in terms of rows of $\mathbold M$.

A property seen in real-world graphs is that they are usually dynamic. This means they frequently obtain/loose nodes or edges. As a result, after an update operation the solution found for the linear graph regression problem must be updated. This should be done in a time much less than the time of computing it from scratch, for the updated graph. In this section, we discuss properties that matrix embeddings suitable for updating the solution of graph regression should satisfy. Before that, using an example we discuss what happens to a matrix embedding, when an update operation occurs in the graph.

Depending on how we define the matrix embedding $\mathbold{M}$, deleting/inserting a node/edge may or may not result in changes in the vector embeddings of the other nodes. As an example, consider Figure 1 that shows a directed graph $G$, a node deletion operation (Figure 1(a)), and a node insertion operation (Figure 1(b)). In this example, the matrix embedding of the graph is defined as follows: for each node $i$ in $G$ we keep a row of size $n$, where the $j^{th}$ entry is the inverse of the distance (shortest path size) from $i$ to node $j$. If there is no directed path from $i$ to $j$ or if $i=j$, the entry will be [math]. In the figure, the row corresponding to each node shows its vector embedding, and the matrix consisting of these rows is the matrix embedding. For example, in the graph on the top figure (graph $G$), the vector embedding of node $1$ consists of the following elements: [math], $1$ and $1/2$. First, consider deleting node $3$ from $G$. Since for each node, we have a row and a column in the matrix embedding, we need to delete the row and column corresponding to node $3$. However, this is not enough as it does not always yield a valid matrix embedding for the updated graph. For example, we need also to change the first element of the second row from $1/2$ to [math]. The reason is that after deleting node $3$ and its incident edges, there will be no path from node $2$ to node $1$. The changes in the updated matrix embedding are depicted with red in Figure 1(a).

Then as shown in Figure 1(b), consider inserting a new node $4$ to $G$. In order to update the matrix embedding with respect to this change, we need to add a column and a row corresponding to this new node. The entries of this row and this column are depicted with red in Figure 1(b). Fortunately, node $4$ and its edge to node $3$ do not change the distances between the existing nodes of the graph. So, rows/columns 1-3 of the matrix embedding do not need to change. It is worth highlighting that after detecting these changes in $\mathbf{M}$, matrix $\mathbf{M}^{{\dagger}}$ must be updated accordingly and the updated $\mathbf{M}^{{\dagger}}$ must be multiplied with $\mathbf{b}$ to form the updated solution.

Definition 1**.**

*Let $\mathbold{M}$ be a $n\times m$ matrix embedding of graph $G$ and $\mathsf{h}$ be a (complexity) function of $n$ and $m$. We say $\mathbold{M}$ is $f$-update-efficient, iff the following conditions are satisfied:222Note that the graph and the information used to find the solution of dynamic graph regression, i.e., the pseudoinverse of matrix $\mathbf{M}$ and vector $\mathbf{b}$, change over time. However, we only care about their values before and after an update operation, as we want to find their values after the update operation, based on their values before the update operation. In order to keep notations as simple as possible, we do not parameterize them by time, rather, we simply use the terms before and after the update operation to distinguish these two situations. *

an edge insertion/deletion or an edge weight change in $G$ result in a rank-$K$* update in $\mathbold{M}$, where $K$ is a constant. More precisely, if $\mathbold{M}$ and $\mathbold{M^{\prime}}$ are correct matrix embeddings before and after one of the above-mentioned update operations in the graph, there exist at most $K$ pairs of vectors $\mathbold{c^{k}}$ and $\mathbold{d^{k}}$ (of proper sizes) such that:*

[TABLE]

We refer to each pair $\mathbold{c^{k}}$ and $\mathbold{d^{k}}$ as a pair of update vectors, and to $\sum_{k=1}^{K}\left(\mathbold{c^{k}}\cdot\mathbold{d^{k}}^{*}\right)$ as the update matrix. 2. 2.

a node insertion in $G$ results in adding one column and/or one row to $\mathbold{M}$ and also at most a rank-$K$* update in $\mathbold{M}$.* 3. 3.

deleting the last node (i.e., the node with the largest id) from $G$ results in deleting one column and/or one row from $\mathbold{M}$ and also at most a rank-$K$* update in $\mathbold{M}$.* 4. 4.

after an update operation in $G$, it is feasible to compute all pairs of update vectors in $O(\mathsf{h}(n,m))$ time. 5. 5.

if in $G$ any node $i$ is permuted with the last node (i.e., with the node that has the largest id), this can be expressed, in $O(\mathsf{h}(n,m))$ time, in terms of a rank-$K$* update in $\mathbold{M}$.*

Sometimes when $\mathsf{h}$ is clear from the context, we simply drop it and use the term update-efficient.

Remark 1**.**

The update-efficient property of matrix embeddings is not closed under matrix addition and matrix subtraction, as they may increase the rank of the matrix.

In Section 5, we use update-efficient matrix embeddings to develop an efficient algorithm for dynamic graph regression. Some well-known matrix embeddings belong to this class of matrix embeddings. In Lemma 1 we show that (weighted) adjacency matrix is a $n$-update efficient matrix embedding. Moreover, in Lemma 2 we show that under some conditions, the other widely used matrix embedding, i.e., the Laplacian matrix, provides a $n$-update efficient matrix embedding, too.

Lemma 1**.**

Assume that $G$ is a simple, weighted and directed graph and its matrix embedding $\mathbold{M}$ is defined as its weighted adjacency matrix. $\mathbold{M}$ is a $n$-update-efficient matrix embedding (i.e., $\mathsf{h}(n)=n$).

Proof.

We show that $\mathbold{M}$ satisfies all the five conditions stated in Definition 1.

When an edge is inserted/deleted between nodes $i$ and $j$ or its weight changes, only the $ij^{th}$ entry of $\mathbold{M}$ is revised. Let $q$ denote the amount of this change in $\mathbold{M}[ij]$, which can be either positive or negative. To express it in terms of a pair of update vectors $\mathbold{c}$ and $\mathbold{d}$, we only need to define e.g., $\mathbold{c}$ as a vector (of size $n$) whose all elements are [math] except the $i^{th}$ element, which is $q$; and $\mathbold{d}$ as a vector (of size $n$) whose all elements are [math], except the $j^{th}$ element which is $1$. Obviously, $\mathbold{c}\cdot\mathbold{d}^{*}$ yields a $n\times n$ matrix whose all elements, except the $ij^{th}$ element, are [math]; and its $ij^{th}$ element is $q$. Therefore, condition (1) of Definition 1 is satisfied. 2. 2.

When a new node $i$ is added to $G$, we add a new column for it in $\mathbold{M}$, which contains the weights of its incoming edges (i.e., if there is an edge from a node $j$ to $i$, we put the weight of this edge in the $j^{th}$ entry of the new column). Also, we add a new row for it in $\mathbold{M}$, which contains the weights of its outgoing edges. Furthermore, in $\mathbold{M}$ we only need to update those entries whose column number or row number are $i$. These entries are updated during column/row addition. Hence, no update vector is required. As a result, condition (2) of Definition 1 is satisfied. 3. 3.

When the last node is deleted from $G$, we delete its corresponding column and row from $\mathbold{M}$. Furthermore, in $\mathbold{M}$ we only need to update those entries whose column number or row number are $i$. These entries are already deleted during column/row deletion. Therefore, no update vector is required and hence, condition (3) of Definition 1 is satisfied. 4. 4.

In the case of edge insertion/deletion and edge weight change, the update vectors $\mathbold{c}$ and $\mathbold{d}$ can be computed in $O(n)$ time. In the case of the other update operations, there is no need for update vectors. As a result, condition (4) of Definition 1 is satisfied. 5. 5.

When in $G$ we permute a node $i$ with the last node, in $\mathbold{M}$ we only need to first exchange the $i^{th}$ column with the last column and then in the resulted matrix, exchange the $i^{th}$ row with the last row. These two changes in $\mathbold{M}$ can be expressed in terms of a rank-$K$ update matrix, as follows. We focus on exchanging the $i^{th}$ column with the last column, as exchanging the $i^{th}$ row with the last row can be done in a similar way. Let $l$ denote the index of the last column. First, note that we may consider exchanging the $i^{th}$ column with the $l^{th}$ column as adding to each entry $ji$ in the $i^{th}$ column the value $-\mathbold{M}[ji]+\mathbold{M}[jl]$; and adding to each entry $jl$ in the $l^{th}$ column the value $-\mathbold{M}[jl]+\mathbold{M}[ji]$. Now let focus on the $i^{th}$ column (a similar procedure can be used for the $l^{th}$ column). We want to express the additions to the $i^{th}$ column in terms of a pair of update vectors $\mathbold{c}$ and $\mathbold{d}$. We can define $\mathbold{c}$ as a vector whose $j^{th}$ entry contains $-\mathbold{M}[ji]+\mathbold{M}[jl]$; and $\mathbold{d}$ as a vector whose all entries, except the $i^{th}$ entry, are [math] and its $i^{th}$ entry is $1$. Clearly, $\mathbold{c}\cdot\mathbold{d}^{*}$ yields a matrix whose $i^{th}$ column includes the values $-\mathbold{M}[ji]+\mathbold{M}[jl]$, for $1\leq j\leq n$, and its other entries are [math]. As a result, exchanging the $i^{th}$ column with the $l^{th}$ column can be done using $2$ rank-$1$ pairs of update vectors. Moreover, computing vectors $\mathbold{c}$ and $\mathbold{d}^{*}$ can be done in $O(n)$ time. Hence, condition (5) of Definition 1 is satisfied.

∎∎

Lemma 2**.**

Suppose that $G$ is a weighted, undirected and bounded-degree graph and its matrix embedding $\mathbold{M}$ is defined as its weighted Laplacian matrix. $\mathbold{M}$ is a $n$-update-efficient matrix embedding333Note that when inserting a new node to a bounded-degree graph, at most a constant (bounded) number of edges are drawn between the new node and existing nodes..

Proof.

Assume that the degrees of the nodes of $G$ are bounded by a constant $C$.

When an edge is inserted/deleted between nodes $i$ and $j$ or its weight changes, the entries $ij$, $ji$, $ii$ and $jj$ of $\mathbold{M}$ might change. For each of these entries, similar to the first case of the proof of Lemma 1, we can express the change in terms of a pair of update vectors $\mathbold{c}$ and $\mathbold{d}$. Hence, all these changes can be stated in terms of an update matrix whose rank is at most $4$ (which is generated by the sum of four rank-$1$ matrices) and as a result, condition (1) of Definition 1 is satisfied. 2. 2.

When a new node $i$ is added to $G$, we add a new column for it in $\mathbold{M}$, whose $j^{th}$ entry is $-q$, where $q$ is the weight of the edge between nodes $i$ and $j$ (if there is no edge between $i$ and $j$, $q$ is [math]). In a similar way, we add a new row for it in $\mathbold{M}$. Furthermore, in $\mathbold{M}$ we need to increase by $1$ each entry $jj$ such that $j$ has an edge to $i$. Since the degrees are bounded by $C$, the number of such revisions is at most $C$. Each such revision, can be expressed by a pair of update vectors $\mathbold{c}$ and $\mathbold{d}$, where the $j^{th}$ entries of these vectors are $1$ and the other entries are [math]. These $C$ (rank-$1$) update vectors yield an update matrix whose rank is at most $C$, as a result, condition (2) of Definition 1 is satisfied. 3. 3.

When the last node of $G$ is deleted, we delete its corresponding column and row from $\mathbold{M}$. Furthermore, in $\mathbold{M}$ we need to decrease by $1$ each entry $jj$ such that $j$ has an edge to $i$. Similar to the case of node addition, this can be done by at most $C$ (rank-$1$) update vectors, where the non-zero element of each vector $\mathbold{c}$ is $-1$, rather than $1$. As a result, condition (3) of Definition 1 is satisfied. 4. 4.

For all the update operations, the update vectors $\mathbold{c}$ and $\mathbold{d}$ can be computed in $O(Cn)=O(n)$ time. As a result, condition (4) of Definition 1 is satisfied. 5. 5.

When in $G$ we permute a node $i$ with the last node, in $\mathbold{M}$ we need to update both $\mathbold{D}$ and $\mathbold{W}$. On the one hand, $\mathbold{W}$ can be updated in a way similar to the permutation case of the proof of Lemma 1 and this can be done in $O(n)$ time. On the other hand, by permutation, weighted degrees of the nodes, except node $i$ and the last node, do not change. Therefore $\mathbold{D}$ can be easily updated by exchanging the entry $ii$ and the entry in the last row and last column of $\mathbold{D}$, which can be done by two pairs of update vectors. This can be done in $O(n)$ time. Hence, condition (5) of Definition 1 is satisfied.

∎∎

Note that weighted Laplacian matrix (as well as unweighted Laplacian matrix), without the mentioned constraint on degrees, are not update-efficient, for any arbitrary complexity function $\mathsf{h}$. The reason is that without the mentioned constraint, node addition/deletion may change the degrees of $\Theta(n)$ nodes, which then may require a rank-$\Theta(n)$ update matrix (or $\Theta(n)$ pairs of update vectors). Nevertheless, if we restrict update operations, we might be able to use Laplacian matrix for the regression of general dynamic graphs. For example, if we limit update operations to edge insertion, edge deletion and edge weight change, it will not be necessary for nodes of the graph to have a bounded degree.

5 Dynamic graph regression under least squares error

In this section, we condition on the existence of update-efficient matrix embeddings and show that the exact optimal solution of $l_{2}$ graph regression can be updated in a time much faster than computing it from scratch.

At the high-level, the algorithm of solving $l_{2}$ dynamic graph regression consists of three phases:

1. the matrix embedding phase, wherein we compute an update-efficient matrix embedding $\mathbold{M}$ for the given graph (we assume that it is static),
2. the pre-processing phase, wherein we assume that we are given a static graph and we find a solution for it, and
3. the update phase, wherein after any update operation in $G$, $\mathbold{M}$ and the already found solution are revised to become valid for the new graph. For the pre-processing phase, we use $\mathbold{M}^{{\dagger}}\cdot\mathbold{b}$ as the optimal solution. Naively computing the SVD and pseudoinverse of $\mathbold{M}$ requires $O\left(\min(nm^{2},mn^{2})\right)$ time. However using fast matrix multiplication [15] (which is not practical!), it can be done in $O(nm^{1.3728})$ time. In the proof of Theorem 1, we discuss how the optimal solution is updated, after an update operation in $G$.

Theorem 1**.**

Let $\mathbold{M}$ be a $n\times m$ update efficient matrix embedding of graph $G$. After a pre-processing phase which takes $O\left(\min(nm^{2},n^{2}m)\right)$ time, after any update operation in the graph, the exact optimal solution of the $l_{2}$ graph regression problem for the updated graph can be computed in $O(nm)$ time.

Proof.

After an update operation in $G$, we require to first update $\mathbold{M}^{{\dagger}}$ and then, update $\mathbold{M}^{{\dagger}}\cdot\mathbold{b}$. The way of updating $\mathbold{M}^{{\dagger}}$ depends on the update operation done in the graph.

• •

Edge insertion/deletion or edge weight change: in any of these cases, due to the update-efficient property of $\mathbold{M}$, we have a sequence of at most $K$ rank-$1$ updates:

[TABLE]

for $1\leq k<K$, where $\mathbold{c^{k}}$ and ${\mathbold{d^{k}}}$ are a pair of update vectors, $\mathbold{M^{1}}=\mathbold{M}$ and $\mathbold{M^{K}}$ is the correct matrix embedding of $G$ after the update operation. After each rank-$1$ update $\mathbold{M^{k+1}}=\mathbold{M^{k}}+\mathbold{c^{k}}\cdot{\mathbold{d^{k}}}^{*}$, we may exploit e.g., the algorithm of Meyer [6] that given a matrix $\mathbold{A}$ and its Moore-Penrose pseudoinverse $\mathbold{A}^{{\dagger}}$ and a pair of update vectors $\mathbold{c}$ and $\mathbold{d}$, computes Moore-Penrose pseudoinverse of $(\mathbold{A}+\mathbold{c}\cdot\mathbold{d}^{*})$. Due to many notations and a long explanation required to introduce this method, we here omit its description and refer the interested reader to [6]. The key point is that given $\mathbold{A}^{{\dagger}}$, computing $(\mathbold{A}+\mathbold{c}\cdot\mathbold{d}^{*})^{{\dagger}}$ can be done in $O(nm)$ time. Therefore and after applying this algorithm for at most $K$ times, we can compute the Moore-Penrose pseudoinverse of the matrix embedding of the updated graph in $O(Knm)=O(nm)$ time.

• •

Node insertion: in this case, we need to follow a two-phase procedure. In the first step, we require to append a row and (if needed) a column to $\mathbold{M}$ that correspond to the new node and carry out its embedding information. Let’s focus on appending a new column (appending a new row can be dealt with in a similar way). Speaking precisely, we have matrix

[TABLE]

where $\mathbold{a}$ is the column corresponding to the new node and $\mathbold{M}$ is the matrix embedding of $G$ before the update operation. We want to compute $\mathbold{M^{\prime}}^{{\dagger}}$ based on $\mathbold{M}^{{\dagger}}$ and $\mathbold{a}$. For this, we may use e.g., the Greville’s algorithm [18], which is as follows. Let $\mathbold{d}=\mathbold{M}^{{\dagger}}\cdot\mathbold{a}$, $\mathbold{c}=\mathbold{a}-\mathbold{M}\cdot\mathbold{d}$, and $\mathbold{\theta}$ be the null matrix (of proper size), i.e., the matrix whose all elements are zero. Then

[TABLE]

where

[TABLE]

Since adding the new node may affect the vector embeddings of the existing nodes by a sequence of at most $K$ rank-$1$ update vectors, in the second step, we need to reflect these changes to the matrix embedding of $G$. In other words, for at most $K$ pairs of vectors $\mathbold{c^{k}}$ and $\mathbold{d^{k}}$, we have:

[TABLE]

where $\mathbold{M^{1}}$ is $\mathbold{M^{\prime}}$ and $\mathbold{M^{K}}$ is the correct matrix embedding of $G$ after the node insertion. Hence, similar to the previous case, we may use the algorithm of Meyer [6] to compute ${\mathbold{M^{k+1}}}^{{\dagger}}$ based on ${\mathbold{M^{k}}}^{{\dagger}}$. Each of these two steps takes $O(nm)$ time.

• •

Node deletion: in this case, we need to follow a three-phase procedure. In the first step, we perform a permutation on $\mathbold{M}$ so that the node that we want to delete becomes the last node in the matrix (i.e., it becomes the node with the largest id). Note that updating $\mathbold{M}$ with this new permutation of nodes may need to call a sequence of at most $K$ rank-$1$ update vectors. Hence, we may need to use Meyer’s algorithm [6] to compute each ${\mathbold{M^{k+1}}}^{{\dagger}}$ based on ${\mathbold{M^{k}}}^{{\dagger}}$ and the update vectors, where ${\mathbold{M^{1}}}=\mathbold{M}$. In the second step, we need to delete a row and (if needed) a column from $\mathbold{M}$ that correspond to the deleted node and carry out its embedding information. Let’s again focus on deleting a column (as deleting a row can be done in a similar way). We have matrix $\mathbold{M^{\prime}}$ such that

[TABLE]

where $\mathbold{a}$ is the column corresponding to the deleted node and $\mathbold{M}$ is matrix embedding of $G$ before the update operation. We may again use the Greville’s algorithm [18] to compute $\mathbold{M^{\prime}}^{{\dagger}}$ based on $\mathbold{M}^{{\dagger}}$ and $\mathbold{a}$. Finally, since deleting a node may change the vector embeddings of the existing nodes by a sequence of at most $K$ rank-$1$ update vectors, in the third step, we need to apply these changes to the matrix embedding of the graph. Hence and similar to the previous cases, we can use Meyer’s algorithm [6] to compute each ${\mathbold{M^{k+1}}}^{{\dagger}}$ based on ${\mathbold{M^{k}}}^{{\dagger}}$ and the update vectors, with ${\mathbold{M^{1}}}=\mathbold{M^{\prime}}$. Each of these three steps takes at most $O(nm)$ time.

As a result, after an update operation in $G$, $\mathbold{M}^{{\dagger}}$ can be updated in $O(nm)$ time. Note that in the case of node deletion, when we perform a permutation on the nodes of $G$ and rows of $\mathbold{M}$, we need also consistently permute the elements of $\mathbold{b}$ and then, remove from $\mathbold{b}$ the measured value of the deleted node (which after the permutation will be the last element of $\mathbold{b}$). These operations can be done in $O(n)$ time. Furthermore, after a node insertion, we need to add the measured value of the new node to the end of $\mathbold{b}$, which can be done in $O(1)$ time. A naive multiplication of the updated $\mathbold{M}^{{\dagger}}$ and the updated $\mathbold{b}$ yields the optimal solution of the updated graph and it takes only $O(nm)$ time. ∎∎

Corollary 1**.**

Given a graph $G$, if its matrix embedding is defined as its (weighted or unweighted) adjacency matrix, after any update operation in $G$ (i.e., node insertion or node deletion or edge insertion or edge deletion or edge weight change), the exact optimal solution of the $l_{2}$ dynamic graph regression problem can be updated in $O(n^{2})$ time.

Proof.

Lemma 1 says that the (weighted) adjacency matrix of $G$ is a $n$-update-efficient and therefore, a $n^{2}$-update-efficient matrix embedding of $G$. Hence and as Theorem 1 says, after any update operation in the graph, the optimal solution of the $l_{2}$ graph regression problem for the updated graph can be computed in $O(n^{2})$ time. ∎∎

To the best of our knowledge, Corollary 1 provides the first result on updating the exact optimal solution of linear regression over general dynamic graphs, in a time less than computing it from scratch. Note that if the weighted adjacency matrix of $G$ is used as the matrix embedding, computing the optimal solution of dynamic graph regression from scratch will take $O(n^{3})$ time.

As mentioned earlier, if we restrict the update operations to edge insertion, edge deletion and edge weight change, the Laplacian matrix will be a $n^{2}$-update-efficient matrix embedding. Hence, we will have the following result.

Corollary 2**.**

Suppose that we are given an undirected graph $G$, whose matrix embedding is defined as its (weighted or unweighted) Laplacian matrix.

After an edge insertion or an edge deletion or an edge weight change in $G$, we can update the exact optimal solution of the $l_{2}$ dynamic graph regression problem in $O(n^{2})$ time. 2. 2.

If $G$ is a bounded-degree graph, after any update operation in $G$, i.e., node insertion (wherein a bounded number of edges are drawn between the new node and the existing nodes) or node deletion or edge insertion or edge deletion or edge weight change, the exact optimal solution can be updated in $O(n^{2})$ time.

In the end, it might be of interest to see whether similar results can be obtained for other regression models. Someone may start with linear regression under least absolute deviation, as it is the closest setting to our studied problem, and then go beyond linear models and study nonlinear regression functions for dynamic graphs. In dynamic graph regression under least absolute deviation, the distance is measured using the $l_{1}$ norm. More precisely, we want to solve the following problem:

[TABLE]

where the $l_{1}$ norm of a vector $\mathbf{v}$ is defined as follows: $||\mathbf{v}||_{1}=\sum_{i}|\mathbf{v}_{i}|$. What makes this problem and most of the other nonlinear regression problems theoretically more challenging, is that they do not have a closed form solution [39], i.e., a solution like what is presented in Equation 2 for our studied problem. However, we believe it might be possible to derive similar results for some of these models, and leave it as an interesting direction for future work.

6 Experimental results

In this section, we empirically evaluate the update algorithm. First in Section 6.1, we report the results of our experiments over synthetic and real-world graphs. Then in Section 6.2, we discuss the applicability and usefulness of dynamic graph regression and its update algorithm, in a real-world case study.

6.1 Empirical evaluation

In this section, we examine the empirical efficiency of updating the solution of dynamic graph regression, compared to computing it from scratch. Since node insertion and node deletion include edge insertion and edge deletion, we only consider these two update operations. In node insertion, a new node along with some edges connecting it to the existing nodes are added to the graph. In node deletion, the node with the largest id is deleted from the graph. Each of these operations is conducted for $50$ times and the average results are reported. We define the matrix embedding of the graph as its adjacency matrix. We run both the update algorithm and the algorithm of computing the solution from scratch, and compare their times.

6.1.1 Synthetic graphs

We study how the update algorithm scales with respect to the size of the graph, compared to the algorithm that computes the solution from scratch. We generate several ER random graphs [14] with $100$, $1250$, $2500$, $3750$, $5000$, $6250$, $7500$, $8750$ and $10000$ nodes, all with the same edge induction probability $0.5$. Figures 2(a) and 2(b) present the results over the generated graphs, respectively for node insertion and node deletion. Due to large differences between update and scratch times, in the charts of these figures the vertical axes are presented in the logarithmic scale. As depicted in the figures, incrementally updating the solution rather than computing it from scratch takes much less time. Moreover, it is more scalable: it can handle large graphs within still an empirically tractable time. The time of computing the solution from scratch quickly increases by increasing the size of the graph, so that performing it for large graphs is intractable in practice.

6.1.2 Real-world graphs

We also evaluate the algorithms over the following real-world graphs: wiki-vote [29]444https://snap.stanford.edu/data/wiki-Vote.html, lastfm-asia [36]555https://snap.stanford.edu/data/feather-lastfm-social.html and soc-sign-bitcoinotc [27]666https://snap.stanford.edu/data/soc-sign-bitcoin-otc.html. Wiki-vote has 7115 nodes and 103689 edges, lastfm-asia has 7624 nodes and 27806 edges, and soc-sign-bitcoinotc has 5881 nodes and 35592 edges. The comparison results are presented in Table 1. They depict that updating the solution of linear regression is considerably more efficient than computing it from scratch.

We note than while wiki-vote and lastfm-asia have almost the same number of nodes (lastfm-asia has slightly more nodes), over wiki-vote both update time and scratch time are much larger. The reason is that wiki-vote is a considerably more dense graph than lastfm-asia. When computing the solution from scratch, we need to compute the pseudoinverse of the adjacency matrix of the updated graph. This operation is much more time consuming for dense graphs than sparse graphs. On the other hand, for dense graphs more pairs of update vectors are induced, after the update operation. Moreover, updating the pseudoinverse of the matrix is more time consuming. As a result, update time over a graph such as wiki-vote is larger than update time over a sparser graph such as lastfm-asia.

6.2 A case study

Cryptocurrency price prediction is a challenging task in market data analysis. Correlations between cryptocurrencies [4] can be useful in predicting cryptocurrency prices. In this section, we investigate applying a linear regression model to predict cryptocurrency prices. From the coinmetrics website777https://charts.coinmetrics.io, we collect the correlation information and the prices (in the US Dollar) of the following cryptocurrencies, in July $28^{th}$ 2021: Bitcoin (BTC), Decred (DCR), DigiByte (DGB), Dogecoin (DOGE), Litecoin (LTC), MCO Token (MCO) and Vertcoin (VTC). Figure 3(a) shows the weighted graph of the correlations, where the nodes are the currencies and the weight of an edge represents the correlation between its two endpoints. Figure 3(b) presents the prices of the cryptocurrencies. Our goal is to develop a linear model that predicts the price of a cryptocurrency, in the form of a linear combination of its correlations with the other cryptocurrencies. Since the correlations and the prices are dynamic and time-variant, we want to update the model by the changes in data.

To do so, we consider the weighted adjacency matrix of the correlation graph of Figure 3(a) as the matrix embedding $\mathbold M$, and the values presented in Figure 3(b) as the vector $\mathbold b$. We solve the corresponding linear regression problem (under $l_{2}$ norm). Its exact optimal solution is:

[TABLE]

This means that in the linear regression, the weight of the correlation with BTC is $-36497$, the weight of the correlation with DCR is $12368$, the weight of the correlation with DGB is $14436$ and so on.

When the correlation graph or vector $\mathbold b$ change over time, we need to accordingly update the weights. We note that as stated in Section 4, weighted adjacency matrix used in this case study is an update efficient matrix embedding. So, the weights of the linear regression can be updated efficiently. An example change is the decrease of the correlation between BTC and DGB from $0.7891$ to $0.7682$. After updating our linear regression model with respect to this change, the weights in vector $\mathbold x$ become:

[TABLE]

7 Conclusion

In this paper, we studied the theory of linear regression over dynamic graphs. First, we presented the class of update-efficient matrix embeddings, that defines conditions sufficient for a matrix embedding to be used for least squares dynamic graph regression. Then, we showed that given a $n\times m$ update-efficient matrix embedding (e.g., the adjacency matrix), after an update operation in the graph, the exact optimal solution of graph regression can be updated in $O(nm)$ time. Finally by conducting experiments over synthetic and real-world graphs, we showed the high empirical efficiency of the update algorithm.