On representation power of neural network-based graph embedding and beyond

TL;DR

This paper investigates the limitations of inner product similarity in neural network-based graph embedding and introduces Shifted IPS (SIPS), a novel model capable of approximating a broader class of similarities, including non-PD and CPD types.

Contribution

The paper proposes SIPS, a new model that extends IPS to approximate any CPD similarity, enhancing the flexibility and applicability of neural network-based graph embeddings.

Findings

SIPS outperforms IPS in numerical experiments.

SIPS can approximate non-PD similarities like negative Wasserstein distance.

Theoretical extension of SIPS to Minkowski space for more general similarities.

Abstract

We consider the representation power of siamese-style similarity functions used in neural network-based graph embedding. The inner product similarity (IPS) with feature vectors computed via neural networks is commonly used for representing the strength of association between two nodes. However, only a little work has been done on the representation capability of IPS. A very recent work shed light on the nature of IPS and reveals that IPS has the capability of approximating any positive definite (PD) similarities. However, a simple example demonstrates the fundamental limitation of IPS to approximate non-PD similarities. We then propose a novel model named Shifted IPS (SIPS) that approximates any Conditionally PD (CPD) similarities arbitrary well. CPD is a generalization of PD with many examples such as negative Poincar\'e distance and negative Wasserstein distance, thus SIPS has a potential impact to significantly improve the applicability of graph embedding without taking great care in configuring the similarity function. Our numerical experiments demonstrate the SIPS's superiority over IPS. In theory, we further extend SIPS beyond CPD by considering the inner product in Minkowski space so that it approximates more general similarities.