Asymptotics of $\ell_2$ Regularized Network Embeddings

TL;DR

This paper analyzes the asymptotic behavior of $\,\ell_2$-regularized network embeddings, revealing their connection to graphon estimation and providing theoretical guarantees for the distribution of learned embeddings.

Contribution

It establishes the asymptotic distribution of $\,\ell_2$-regularized embeddings and links stochastic gradient subsampling to nuclear-norm penalties in graphon estimation.

Findings

Regularization leads to graphon estimation with nuclear-norm penalty

Asymptotic distribution of embeddings is characterized

Empirical results show covariate concatenation improves performance

Abstract

A common approach to solving prediction tasks on large networks, such as node classification or link prediction, begin by learning a Euclidean embedding of the nodes of the network, from which traditional machine learning methods can then be applied. This includes methods such as DeepWalk and node2vec, which learn embeddings by optimizing stochastic losses formed over subsamples of the graph at each iteration of stochastic gradient descent. In this paper, we study the effects of adding an $\ell_2$ penalty of the embedding vectors to the training loss of these types of methods. We prove that, under some exchangeability assumptions on the graph, this asymptotically leads to learning a graphon with a nuclear-norm-type penalty, and give guarantees for the asymptotic distribution of the learned embedding vectors. In particular, the exact form of the penalty depends on the choice of subsampling method used as part of stochastic gradient descent. We also illustrate empirically that concatenating node covariates to $\ell_2$ regularized node2vec embeddings leads to comparable, when not superior, performance to methods which incorporate node covariates and the network structure in a non-linear manner.