DeeWaNA: An Unsupervised Network Representation Learning Framework Integrating Deepwalk and Neighborhood Aggregation for Node Classification

TL;DR

DeeWaNA is a new unsupervised framework that combines random walks and neighborhood aggregation to improve node classification in networks.

Contribution

The novel integration of DeepWalk and attention-based neighborhood aggregation into a unified model for network representation learning.

Findings

DeeWaNA outperforms four existing unsupervised methods in node classification tasks.

The attention-based weighting mechanism improves neighborhood relationship modeling.

The framework enhances structural feature extraction and representation quality.

Abstract

This paper introduces DeeWaNA, an unsupervised network representation learning framework that unifies random walk strategies and neighborhood aggregation mechanisms to improve node classification performance. Unlike existing methods that treat these two paradigms separately, our approach integrates them into a cohesive model, addressing limitations in structural feature extraction and neighborhood relationship modeling. DeeWaNA first leverages DeepWalk to capture global structural information and then employs an attention-based weighting mechanism to refine neighborhood relationships through a novel distance metric. Finally, a weighted aggregation operator fuses these representations into a unified low-dimensional space. By bridging the gap between random-walk-based and neural-network-based techniques, our framework enhances representation quality and improves classification accuracy. Extensive evaluations on real-world networks demonstrate that DeeWaNA outperforms four widely used unsupervised network representation learning methods, underscoring its effectiveness and broader applicability.

Figures

Full text

1. Introduction

Network Representation Learning (NRL) aims to learn latent, low-dimensional network representational vectors [1]. With information networks becoming ubiquitous in describing the complex relationships in large real-world applications in the forms of social networks, biological networks, and citation networks, NRL has played a crucial role in many network analysis applications, including node classification, link prediction, community detection, and subnetwork similarity [1,2,3,4]. Because of the advantages of adaption for the absence of expensive manual annotation data, many Unsupervised Network Representation Learning (UNRL) methods [5,6,7,8,9,10] have been proposed and have shown remarkable achievements.

DeepWalk is one of the widely used random-walk-based UNRL algorithms, due to its few-shot friendliness and simple distributed computation. It is based on context sampling of node sequences on networks to calculate representation vectors, which results in a reduced capacity for capturing neighborhood messages. Many researches have demonstrated the importance of catching neighborhood messages in network representation learning, and various neighborhood aggregation methods have been proposed [11,12,13,14]. Integrating DeepWalk with neighborhood aggregation techniques appears to be a promising strategy that provides representational vectors with improved performance for machine learning models in downstream network analysis tasks.

Although the existing aggregation methods differ from model to model [15], the distances between nodes and their neighbors in embedding space are not taken into account by the approaches. The updated representational vectors may cause tasks to become more challenging when neighborhood nodes that are located at varying distances from a central node are considered equally throughout neighborhood aggregation operators. We illustrate an example of a node classification task, as shown in Figure 1. Given a network G, as shown in Figure 1a, neighborhood aggregation techniques can reduce the difficulty of the classification task from the perspective of position in the embedding space, as shown in Figure 1b. In certain circumstances, poor aggregation techniques will arrange neighbor nodes closely together in the embedding space, as shown in Figure 1c. In some awful circumstances, catastrophic aggregation techniques might make tasks more difficult, as shown in Figure 1d. As a result, the neighbors should be distinguished by distinct weights throughout the aggregation process. Many works have used the attention mechanism, a weighting technique, to discriminate between neighbors [10,16].

In this paper, we presented the DeeWaNA framework for unsupervised network representation learning, which combines DeepWalk with neighborhood aggregation. Specifically, DeeWaNA initially makes use of DeepWalk to obtain the nodes’ self-representational vectors. Then, the framework employs a weighted neighborhood aggregation operation to produce local representational vectors of nodes based on their self-representational vectors. To distinguish between the various effects of neighbor nodes on their center node, DeeWaNA uses the self-attention between nodes and their one-hop neighbor nodes as the weights in the weighted neighborhood aggregation process. For the purpose of obtaining self-attention, we suggest a Euclidean distance-based learning task on representational vectors in embedding space. DeeWaNA fuses the sequential connecting self-representational vectors and local representational vectors into a single unified embedding space using the Principal Component Analysis (PCA) method [17].

We conducted thorough tests to evaluate DeWaNA against four widely used unsupervised network representation learning algorithms on a range of four real-world datasets for the single-label/multi-label node classification tasks. Our experiments demonstrate that DeeWaNA can provide the network representational with enhanced performance to machine learning models on single-label/multi-label node classification tasks.

2. Related Work

It is common practice to employ random-walk-based approaches for unsupervised network representation learning [18,19]. Deepwalk is based on a strategy of choosing neighbors with equal probability for sampling node sequences that are fed into the skip-gram model for network embedding [6]. The node2vec algorithm is an improved version of DeepWalk, which utilizes two parameters for adjusting the usage of a deep-first search and a breadth-first search in sampling node sequences [9]. The off-the-shelf random-walk-based approach, however, is unable to fully utilize the neighbors’ message that is frequently employed in deep-structure-based network representation learning models.

Many deep-structure-based models have been proposed for network representation learning [5,11,12]. Neighborhood aggregation plays an important role in neural-network-based models for learning network representational vectors with high performance [5,13,15,20,21,22]. These studies do not take into account the various effects of neighbors on aggregation operators. By utilizing masked self-attentional stacking layers, which may implicitly assign various weights to different nodes in a neighborhood without expensive matrix operation or knowing graph topology beforehand, Graph Attention Networks (GATs) [10] are utilized to attend to the impacts of neighbors. Although GATs can discriminate between various neighbors, they need labels to acquire self-attention meaning; hence, designing learning tasks is necessary when employing GATs in unsupervised network representation learning.

3. Methodology

In this section, we offer a framework called DeeWaNA that combines DeepWalk with weighted neighborhood aggregation for unsupervised network representation learning. DeeWaNA incorporates weighted neighborhood aggregation into DeepWalk to improve previous random-walk-based unsupervised network representation learning techniques that are unable to collect neighbor information. Consequently, by utilizing DeepWalk and the weighted neighborhood aggregation technique to gather neighborhood information, DeeWaNA can produce high-performance network representation.

3.1. The General Framework of DeeWaNA

We first present the general framework of DeeWaNA, which incorporates DeepWalk and weighted neighborhood aggregation techniques and is illustrated in Figure 2.

First, given a network G, where V is the set of nodes and E is the set of edges, we use DeepWalk, defined as $[eqn]$ , to obtain the representational vectors of nodes in V. The $[eqn]$ is the self-representational vector of node v, which is calculated as follows:

$$[eqn]$$

Then, the nodes’ self-representational vectors are utilized to calculate the Euclidean distance between the nodes and their neighbors in embedding space, which serves as the aim of learning self-attention. By using weighted neighborhood aggregation, the local representational vectors of nodes are created from their neighbors’ self-representational vectors. Finally, the sequentially connected vectors of the self-representational vectors and the local representational vectors are fused into a single unified embedding space by Principle Component Analysis (PCA).

3.2. Weighted Neighborhood Aggregation

The process of neighborhood aggregation is frequently used to collect neighborhood information from adjacent nodes. The neighborhood information of node u is embedded as the local representational vector $[eqn]$ . The $[eqn]$ is calculated as follows:

$$[eqn]$$

where $[eqn]$ is the set of the neighbors of the node u, and $[eqn]$ is the self-representational vector of node v. DeeWaNA calculates the weights between nodes and incorporates the weights into the neighborhood aggregation to distinguish the influences of neighbor nodes on the central node. The local representational vector of node u, using weighted neighborhood aggregation, is calculated as follows:

$$[eqn]$$

where $[eqn]$ represents the set of neighbors of node u, $[eqn]$ is the weight between node u and node v, and $[eqn]$ is the self-representational vector of node v. The next section presents three distinct weighting calculations.

3.3. Euclidean-Distance-Based Weighting Calculations

Self-attention is a widely used learning attention method to weight the various relationship between nodes and their neighbors [23]. To obtain the weights between nodes and their neighbors, we proposed three Euclidean-distance-based weighting calculations.

3.3.1. One-Head Self-Attention

The self-representational vectors of nodes and the vectors of their neighbors are used as inputs in the one-head self-attention model to determine the attention (weights), as shown in Figure 3. The attention function is described as follows:

$$[eqn]$$

where $[eqn]$ , $[eqn]$ , and $[eqn]$ ; $[eqn]$ is the matrix of representation, the learning matrices are $[eqn]$ , $[eqn]$ is the size of inputs, and $[eqn]$ is the dimension of representation. The outputs are put into a full-connect Multi-Layer Perception (MLP). The MLP consists of three layers with ( $[eqn]$ ) neurons. The estimated weights of node i by one-head self-attention are calculated as follows:

$$[eqn]$$

We utilize the distance in embedding space d between nodes as the target of the learning task. The output of the MLP is the estimated distance $[eqn]$ . We aim to minimize the loss function as follows:

$$[eqn]$$

for training the self-attention model.

3.3.2. Multi-Head Self-Attention

In contrast to the one-head self-attention model with queries (Q), keys (K), and values (V), multi-head self-attention attends to information from various representational subspaces, as depicted in Figure 4. Each subspace of representation is processed by the one-head self-attention function. The calculation of multi-head self-attention is described as follows:

$$[eqn]$$

where $[eqn]$ , $[eqn]$ , $[eqn]$ $[eqn]$ $[eqn]$ . Each subrepresentation is calculated by Equation (5). The value of the multi-head self-attention is calculated as follows:

$$[eqn]$$

For the MLP structure, there are three layers with ( $[eqn]$ , $[eqn]$ , $[eqn]$ ) neurons ( $[eqn]$ ).

3.3.3. Non-Learning Weighting Calculation

Self-attention demands unacceptably long computation times and memory requisition on some large networks. To solve this issue, a straightforward weighting calculation is suggested and illustrated in Figure 5. The distance $[eqn]$ between node u and node v is normalized into the (0, 1) range as follows:

$$[eqn]$$

The weight between node v and node u is calculated by the following:

$$[eqn]$$

where $[eqn]$ is the set of neighbors of node u. The non-learning approach substitutes the self-attentional method to calculate the weights between nodes when a node has a large number of neighbors.

3.4. Connection and Fusion

DeeWaNA fuses the linked representational vectors into a single unified embedding space by PCA after splicing the local representational vectors after the self-representational vectors. This is because the simple connection of two different-view representational vectors may cause adverse effects on following network analysis tasks due to the representational vectors coming from different domains.

4. Experiments

We evaluated DeeWaNA on four real-world networks and compared it against four widely used unsupervised network representation learning algorithms, including (DeepWalk [6], LINE [7], Node2vec [9], and SDNE [8]). We set the dimension of representation in all the methods as 128 and used the two-step representation in the LINE. The restricted number of neighbors was set to 10 in all simplified versions. We set the number of heads in multi-head self-attention to 4. Other parameters were set as default in corresponding papers. All experiments were run on the platform of 2.8 GHz 4-core Intel i7, 16 GB 2133 MHz LPDDR3.

4.1. Datasets

We employed four real-world datasets (Blogcatalog [24], Protein-Protein Interactions (PPI) [25], Amherst, Mich [26]) to evaluate the performance of DeeWaNA, and the detailed statistics of these datasets are listed in Table 1.

4.2. Experimental Results

We utilized a one-vs-rest logistic regression model [6] to evaluate the quality of the learned representational vectors of all methods in node classification tasks. The training set was created by randomly selecting a proportion of the nodes, and the testing set was created using the remaining nodes. We repeated this process 10 times to calculate the average values on the Micro-F1.

4.2.1. Results on Multi-Label Classification Tasks

For all multi-label networks, DeeWaNA steadily performed better than other competitors within the rate of training data from 0.2 to 0.8. The experimental results of the multi-label classification tasks on Blogcatalog and PPI are displayed in Table 2 and Table 3.

On the Blogcatalog dataset, DeeWaNA with the non-learning weighting calculation produced the best performance (bold indicated) across all training proportions. One-head/multi-head self-attention was unable to produce the desired results due to memory limitations, as indicated by ‘-’. On the PPI dataset, DeeWaNA performed best at training proportions of (0.2, 0.8) when using the non-learning weighting calculation and at (0.4, 0.6) when using a combination of the multi-head self-attention and the non-learning calculation.

4.2.2. Results on Single-Label Classification Tasks

For all single-label networks, DeeWaNA consistently outperformed its competitors in the 0.2 to 0.8 range of training data proportions. The experimental results of single-label node classification tasks on Amherst and Mich are displayed in Table 4 and Table 5. When one-head self-attention and the non-learning calculation were coupled, DeeWaNA fared the best (bold indicated) in all training proportions on the Amherst dataset. On the Mich dataset, when utilizing the one-head self-attention and the non-learning calculation, DeeWaNA scored best at the training proportions of (0.4, 0.6, and 0.8) and at (0.2) when using the multi-head self-attention and the non-learning weighting calculation.

Our findings show that the various weighting calculations for DeeWaNA perform differently in the multi-label networks, while the combination of the one-head self-attention and the non-learning weighting calculation performs well in the majority of situations in the single-label networks. In single-label tasks, we advise utilizing one-head self-attention along with a non-learning weighting calculation, and in multi-label tasks, choose an appropriate weighting calculation.

4.2.3. Study on Effectiveness of Weighted Neighborhood Aggregation

To demonstrate the effectiveness of the weighted neighborhood aggregation, we conducted an experimental investigation applying weighted neighborhood aggregation to three widely used unsupervised network representation learning methods. LiNA is the technique that uses LINE instead of DeepWalk in DeeWaNA. NoNA is the method that substitutes the node2vec algorithm for DeepWalk in DeeWaNA. SeNA is the approach that replaces DeepWalk in DeeWaNA with the SDNE method. In this part, all methods utilized non-learning weighting calculation. The experimental results are displayed in Table 6.

The experimental results showed that, with the exception of the LINE algorithm on PPI and the node2vec method on Amherst at the 0.2 training proportion, the weighted neighborhood aggregation could improve the corresponding unsupervised learning algorithm in all cases. We speculate that the neighborhood aggregation amplifies the noise in the poor-quality self-representational vectors obtained by node2vec and SDNE in the low training proportion. In addition, comparing DeeWaNA with LiNA, NoNA, and SeNA, as described in Table 7, applying DeepWalk in DeeAWaNA was the best option for obtaining self-representational vectors for all cases except on the Mich dataset. Even though NoNA outperformed other algorithms on the Mich dataset in terms of the (0.2, 0.4, 0.8) training proportions, we still favor DeepWalk due to the node2vec’s good performance in NoNA depending on hyper-parameter tuning.

4.2.4. Study on Relationship Between Distance and Weights

To explore the relationship between distance and weights, we compared two kinds of non-learning weighting calculations in DeeWaNA on multi-/single-label classification tasks. DeeWaNA processed the weights being inversely proportional to distances. The calculation is as defined in Equation (9). DeeWaNA (F) processed the weights being proportional to distances. In DeeWaNA (F), the distance $[eqn]$ between node u and node v is normalized into the (0, 1) range as follows:

$$[eqn]$$

All DeeWaNA versions employed the non-learning weighting calculation in this part. The experimental results are displayed in Table 8. We found that the DeeWaNA (N) obtained the dominant position in all cases. This is also in line with our intuition that the farther the distance, the less the influence.

4.2.5. Study on Effectiveness of Features

We carried out an experimental investigation with two DeeWaNA variants on the Blogcatalog dataset to demonstrate the effectiveness of the local representational vectors. DeeWaNA (local) applied the local representational vectors as the final representations. DeeWaNA (cnt) regarded the unfused connection of self-representational vectors and local representational vectors as the final representations. All DeeWaNA versions employed the non-learning weighting calculation in this investigation. The experimental results are listed in Table 9. Comparing the ‘DeepWalk’ and the ‘DeeWaNA (local)’ results, the local-representational vectors have less expressive ability than the self-representational vectors. Comparing the ‘DeepWalk’ and the ‘DeeWaNA (cnt)’ results, the unfused connected representational vectors are more expressive than the self-representational vectors. Comparing the ‘DeeWaNA (cnt)’ and the ‘DeeWaNA (N)’ results, the performance of representation could be enhanced by fusing the linked representational vectors into a single unified embedding space using principal component analysis. The ablation experiment demonstrated the effectiveness of local representational vectors as assistants and the usefulness of the fusion process.

4.2.6. Study on Fusion

We compared DeeWaNA against two variants of DeeWaNA, which are DeeWaNA (AE) and DeeWaNA (ICA). In this part, all of DeeWaNA versions applied non-learning weighting calculation. The DeeWaNA (AE) consists of a five-layer (256-192-128-192-256) encoder–decoder neural network model as the fusion part. The DeeWaNA (ICA) used the Independent Component Analysis algorithm [27] as the fusion part. The experimental results are displayed in Table 10, and the best results are shown in bold. The results demonstrate that the PCA as the fusion process in DeeWaNA could obtain the best performance.

4.2.7. Study on Multi-Head Self-Attention

We investigated the 2/4/8 heads (MH-2/4/8) and the combined multi-head self-attention 2/4/8 heads with non-learning weighting calculations (MH-2L/4L/8L) to ascertain the number of heads in the multi-head self-attention models. The experimental results are displayed in Figure 6. Despite the fact that no particular number of heads predominated over others in all training proportions of the multi-head self-attention models, through the comparison of the stability and the worst case of prediction, we set the multi-head to four in this paper.

4.3. Limitations

Overall, the DeeWaNA could effectively obtain network representational vectors with good performance on the single-label and multi-label node classification tasks. The weighted neighborhood aggregation can easily be applied to improve some unsupervised network representation learning methods. But, there are some limitations in this work. First, for simplified calculations, only one-hop neighbors’ impacts were considered when DeeWaNA processed weighted neighborhood aggregation. The loss of ignoring multi-hop aggregation will be discussed in future works. Second, some high time-consuming or high-memory-required unsupervised network representation learning methods are not included in this work due to our hardware performance limitations. Third, more research is required before claiming weighted neighborhood aggregation can be applied generally, because this work lacks some SOTA unsupervised network representation learning methods that are particularly time- or memory-intensive. We think that despite many limitations, it is still a good attempt to give existing non-deep-structure-based unsupervised representation learning algorithms the ability to capture neighbor information.

5. Conclusions

In this study, we propose DeeWaNA, an unsupervised network representation learning framework that synergistically integrates DeepWalk with neighborhood-aware aggregation for enhanced node classification. The framework’s effectiveness stems from three key innovations: First, DeeWaNA enhances the conventional DeepWalk approach by incorporating neighborhood information awareness, thereby improving the discriminative power of learned node representations. Second, we introduce an adaptive weighted aggregation mechanism that dynamically quantifies heterogeneous neighbor influences during feature integration. Third, we develop a geometry-aware learning paradigm using Euclidean-distance-based self-attention to preserve spatial relationships in the embedding space.

Through comprehensive experimental evaluation on multiple real-world datasets across different domains, we demonstrate DeeWaNA’s superior performance against four unsupervised network representation learning baselines. The consistent outperformance across evaluation metrics reveals that DeeWaNA generates more semantically meaningful representations that significantly boost downstream node classification accuracy compared to existing approaches.