FeatureNorm: L2 Feature Normalization for Dynamic Graph Embedding

TL;DR

This paper introduces FeatureNorm, a simple L2 normalization technique for dynamic graph embedding that prevents feature shrinking and oversmoothing, improving the quality of node representations over time.

Contribution

It proposes a novel L2 feature normalization method to address oversmoothing in dynamic graph embedding, enhancing node distinguishability in evolving graphs.

Findings

FeatureNorm effectively prevents feature shrinking and oversmoothing.

The method outperforms baseline models on four real-world datasets.

Nodes maintain better distinguishability over time with FeatureNorm.

Abstract

Dynamic graphs arise in a plethora of practical scenarios such as social networks, communication networks, and financial transaction networks. Given a dynamic graph, it is fundamental and essential to learn a graph representation that is expected not only to preserve structural proximity but also jointly capture the time-evolving patterns. Recently, graph convolutional network (GCN) has been widely explored and used in non-Euclidean application domains. The main success of GCN, especially in handling dependencies and passing messages within nodes, lies in its approximation to Laplacian smoothing. As a matter of fact, this smoothing technique can not only encourage must-link node pairs to get closer but also push cannot-link pairs to shrink together, which potentially cause serious feature shrink or oversmoothing problem, especially when stacking graph convolution in multiple layers or steps. For learning time-evolving patterns, a natural solution is to preserve historical state and combine it with the current interactions to obtain the most recent representation. Then the serious feature shrink or oversmoothing problem could happen when stacking graph convolution explicitly or implicitly according to current prevalent methods, which would make nodes too similar to distinguish each other. To solve this problem in dynamic graph embedding, we analyze the shrinking properties in the node embedding space at first, and then design a simple yet versatile method, which exploits L2 feature normalization constraint to rescale all nodes to hypersphere of a unit ball so that nodes would not shrink together, and yet similar nodes can still get closer. Extensive experiments on four real-world dynamic graph datasets compared with competitive baseline models demonstrate the effectiveness of the proposed method.