Improving Attention Mechanism in Graph Neural Networks via Cardinality Preservation

TL;DR

This paper provides a theoretical analysis of attention-based GNNs' limitations in distinguishing graph structures due to ignoring cardinality, and proposes CPA models to enhance their discriminative power, validated by experiments.

Contribution

It identifies the failure cases of attention GNNs related to cardinality ignorance and introduces CPA models to preserve cardinality, improving expressiveness.

Findings

Attention GNNs can fail to distinguish certain structures due to cardinality issues.

CPA models improve the discriminative capacity of attention-based GNNs.

Experimental results show CPA models outperform baseline attention GNNs.

Abstract

Graph Neural Networks (GNNs) are powerful to learn the representation of graph-structured data. Most of the GNNs use the message-passing scheme, where the embedding of a node is iteratively updated by aggregating the information of its neighbors. To achieve a better expressive capability of node influences, attention mechanism has grown to be popular to assign trainable weights to the nodes in aggregation. Though the attention-based GNNs have achieved remarkable results in various tasks, a clear understanding of their discriminative capacities is missing. In this work, we present a theoretical analysis of the representational properties of the GNN that adopts the attention mechanism as an aggregator. Our analysis determines all cases when those attention-based GNNs can always fail to distinguish certain distinct structures. Those cases appear due to the ignorance of cardinality information in attention-based aggregation. To improve the performance of attention-based GNNs, we propose cardinality preserved attention (CPA) models that can be applied to any kind of attention mechanisms. Our experiments on node and graph classification confirm our theoretical analysis and show the competitive performance of our CPA models.

Tables

Table 1. Table 1: Testing accuracies(%) of GAT variants (the original GAT and the GAT applied with each of our 4 CPA models) on TRIANGLE-NODE dataset for node classification. We highlight the result of the best performed model. The proportion P 𝑃 P of multisets that hold the properties in Theorem 1 among all multisets is also reported.

Dataset	TRIANGLE-NODE
$P(\%)$	29.2
Original	78.40 $\pm$ 7.65
Additive	91.31 $\pm$ 1.19
Scaled	91.38 $\pm$ 1.23
f-Additive	91.18 $\pm$ 1.24
f-Scaled	91.36 $\pm$ 1.26

Table 2. Table 2: Testing accuracies(%) of GAT-GC variants (the original one and the ones applied with each of our 4 CPA models) on social network datasets. We highlight the result of the best performed model per dataset. The proportion P 𝑃 P of multisets that hold the properties in Theorem 1 among all multisets is also reported for each dataset.

Datasets	RE-B	RE-M5K
$P(\%)$	100.0	100.0
Original	50.00 $\pm$ 0.00	20.00 $\pm$ 0.00
Additive	93.07 $\pm$ 1.82	57.39 $\pm$ 2.09
Scaled	92.36 $\pm$ 2.27	56.76 $\pm$ 2.26
f-Additive	93.05 $\pm$ 1.87	56.43 $\pm$ 2.38
f-Scaled	92.57 $\pm$ 2.06	57.22 $\pm$ 2.20

Table 3. Table 3: Testing accuracies(%) of GAT-GC variants (the original one and the ones applied with each of our 4 CPA models) on bioinformatics datasets. We highlight the result of the best performed model per dataset. The highlighted results are significantly higher than those from the corresponding Original model under paired t-test at significance level 5 % percent 5 5\% . The proportion P 𝑃 P of multisets that hold the properties in Theorem 1 among all multisets is also reported for each dataset.

Datasets	MUTAG	PROTEINS	ENZYMES	NCI1
$P(\%)$	56.9	29.3	29.4	43.3
Original	84.96 $\pm$ 7.65	75.64 $\pm$ 3.96	58.08 $\pm$ 6.82	80.29 $\pm$ 1.89
Additive	89.75 $\pm$ 6.39	76.61 $\pm$ 3.80	58.90 $\pm$ 6.96	81.92 $\pm$ 1.89
Scaled	89.65 $\pm$ 7.47	76.44 $\pm$ 3.77	58.35 $\pm$ 6.97	82.18 $\pm$ 1.67
f-Additive	90.34 $\pm$ 6.05	76.60 $\pm$ 3.91	59.80 $\pm$ 6.18	81.96 $\pm$ 2.01
f-Scaled	90.44 $\pm$ 6.44	76.81 $\pm$ 3.77	58.45 $\pm$ 6.35	82.28 $\pm$ 1.81

Table 4. Table 4: Testing accuracies(%) for graph classification. We highlight the result of the best performed model for each dataset. Our GAT-GC (f-Scaled) model achieves the top 2 on all 6 datasets.

	Datasets	MUTAG	PROTEINS	ENZYMES	NCI1	RE-B	RE-M5K
Baselines	WL	82.05 $\pm$ 0.36	74.68 $\pm$ 0.49	52.22 $\pm$ 1.26	82.19 $\pm$ 0.18	81.10 $\pm$ 1.90	49.44 $\pm$ 2.36
	PSCN	88.95 $\pm$ 4.37	75.00 $\pm$ 2.51	-	76.34 $\pm$ 1.68	86.30 $\pm$ 1.58	49.10 $\pm$ 0.70
	DGCNN	85.83 $\pm$ 1.66	75.54 $\pm$ 0.94	51.00 $\pm$ 7.29	74.44 $\pm$ 0.47	76.02 $\pm$ 1.73	48.70 $\pm$ 4.54
	GIN	89.40 $\pm$ 5.60	76.20 $\pm$ 2.80	-	82.70 $\pm$ 1.70	92.40 $\pm$ 2.50	57.50 $\pm$ 1.50
	CapsGNN	86.67 $\pm$ 6.88	76.28 $\pm$ 3.63	54.67 $\pm$ 5.67	78.35 $\pm$ 1.55	-	52.88 $\pm$ 1.48
	GAT-GC (f-Scaled)	90.44 $\pm$ 6.44	76.81 $\pm$ 3.77	58.45 $\pm$ 6.35	82.28 $\pm$ 1.81	92.57 $\pm$ 2.06	57.22 $\pm$ 2.20

Table 5. Table 5: Dataset Description

Datasets	Graphs	Classes	Features	Node Avg.	Edge Avg.
MUTAG	188	2	7	17.93	19.79
PROTEINS	1113	2	4	39.06	72.81
ENZYMES	600	6	6	32.63	62.14
NCI1	4110	2	23	29.87	32.30
RE-B	2000	2	-	429.63	995.51
RE-M5K	4999	5	-	508.52	1189.75

Equations

$$\displaystyle h_{i}^{l}=\phi^{\,l}\big{(}h_{i}^{l-1},\left\{h_{j}^{l-1},\forall j\in\mathcal{N}(i)\right\}\big{)},$$

$$\displaystyle h_{G}=g\big{(}\left\{h_{i}^{L},\forall i\in G\right\}\big{)}.$$

$$\displaystyle e_{ij}^{l-1}=Att\big{(}h_{i}^{l-1},h_{j}^{l-1}\big{)},$$

$$\displaystyle\alpha_{ij}^{l-1}=\operatorname{softmax}\big{(}e_{ij}^{l-1}\big{)}=\frac{\exp(e_{ij}^{l-1})}{\sum_{k\in\tilde{\mathcal{N}}(i)}\exp\big{(}e_{ik}^{l-1}\big{)}},$$

$$\displaystyle h_{i}^{l}=f^{l}\Big{(}\sum\nolimits_{j\in\tilde{\mathcal{N}}(i)}\alpha_{ij}^{l-1}h_{j}^{l-1}\Big{)},$$

$$\displaystyle h_{i}^{l}=f^{l}\Big{(}\sum\nolimits_{j\in\tilde{\mathcal{N}}(i)}\alpha_{ij}^{l-1}h_{j}^{l-1}+w^{l}\odot\sum\nolimits_{j\in\tilde{\mathcal{N}}(i)}h_{j}^{l-1}\Big{)},$$

$$\displaystyle h_{i}^{l}=f^{l}\Big{(}\psi^{l}\big{(}\big{|}\tilde{\mathcal{N}}(i)\big{|}\big{)}\odot\sum\nolimits_{j\in\tilde{\mathcal{N}}(i)}\alpha_{ij}^{l-1}h_{j}^{l-1}\Big{)},$$

$$\displaystyle h_{i}^{l}=f^{l}\Big{(}\sum\nolimits_{j\in\tilde{\mathcal{N}}(i)}(\alpha_{ij}^{l-1}+1)h_{j}^{l-1}\Big{)},$$

$$\displaystyle h_{i}^{l}=f^{l}\Big{(}\big{|}\tilde{\mathcal{N}}(i)\big{|}\cdot\sum\nolimits_{j\in\tilde{\mathcal{N}}(i)}\alpha_{ij}^{l-1}h_{j}^{l-1}\Big{)}.$$

$$h(c_i,X_i)=\sum _{x\in X_i}{\alpha }_{cxi}f(x),i\in \{1,2\},$$

$$h(c_1,X_1)=h(c,S,\mu )=\sum _{s\in S}\mu (s){\alpha }_{cs1}f(s),$$

$$h(c_2,X_2)=h(c,S,k\cdot \mu )=\sum _{s\in S}k\cdot \mu (s){\alpha }_{cs2}f(s),$$

$$\sum _{s\in S}\mu (s){\alpha }_{cs1}f(s)$$

$$=\frac{{\sum }_{s\in S}\mu (s)\exp (e_{cs1})}{{\sum }_{x\in X_1}\exp (e_{cx1})}f(s),$$

$$\sum _{s\in S}k\cdot \mu (s){\alpha }_{cs2}f(s)$$

$$=k\cdot \frac{{\sum }_{s\in S}\mu (s)\exp (e_{cs2})}{{\sum }_{x\in X_2}\exp (e_{cx2})}f(s),$$

$$\sum _{x\in X_1}\exp (e_{cx})=\frac{1}{k}\sum _{x\in X_2}\exp (e_{cx}).$$

$$\frac{{\sum }_{s\in S}\mu (s)\exp (e_{cs1})}{{\sum }_{x\in X_1}\exp (e_{cx1})}f(s)$$

$$=k\cdot \frac{{\sum }_{s\in S}\mu (s)\exp (e_{cs2})}{{\sum }_{x\in X_2}\exp (e_{cx2})}f(s).$$

$$h(c_1,X_1)$$

$$=k\cdot \frac{{\sum }_{s\in S}\mu (s)\exp (e_{cs2})}{{\sum }_{x\in X_2}\exp (e_{cx2})}f(s)$$

$$=h(c_2,X_2).$$

$$\sum _{x\in X_1}{\alpha }_{cx1}f(x)=\sum _{x\in X_2}{\alpha }_{cx2}f(x),\forall f,Att,$$

$$\sum _{s\in S_1}{\mu }_1(s){\alpha }_{cs1}f(s)=\sum _{s_{\in }{S}_2}{\mu }_2(s){\alpha }_{cs2}f(s),\forall f,Att,$$

$$\sum _{s\in S_1\cap S_2}$$

$$\sum _{s\in S_1\setminus S_2}$$

$$f_1(s)=f_2(s),\forall s\in S_1\cap S_2,$$

$$f_1(s)=f_2(s)-1,\forall s\in S_1\setminus S_2,$$

$$f_1(s)=f_2(s)+1,\forall s\in S_2\setminus S_1.$$

$$\sum _{s\in S_1\cap S_2}$$

$$\sum _{s\in S_1\setminus S_2}$$

$$\displaystyle\sum_{s\in S_{1}\cap S_{2}}\big{(}\mu_{1}(s)\alpha_{cs1}-\mu_{2}(s)\alpha_{cs2}\big{)}f_{2}(s)+$$

$$\sum _{s\in S_1\setminus S_2}{\mu }_1(s){\alpha }_{cs1}(f_2(s)-1)-$$

$$\sum _{s\in S_2\setminus S_1}{\mu }_2(s){\alpha }_{cs2}(f_2(s)+1)=0.$$

$$\displaystyle\sum_{s\in S_{1}\cap S_{2}}\big{(}\mu_{1}(s)\alpha_{cs1}-\mu_{2}(s)\alpha_{cs2}\big{)}f_{2}(s)+$$

$$\sum _{s\in S_1\setminus S_2}{\mu }_1(s){\alpha }_{cs1}{f}_2(s)-\sum _{s\in S_2\setminus S_1}{\mu }_2(s){\alpha }_{cs2}{f}_2(s)=$$

$$\sum _{s\in S_1\setminus S_2}{\mu }_1(s){\alpha }_{cs1}+\sum _{s\in S_2\setminus S_1}{\mu }_2(s){\alpha }_{cs2}$$

$${\mu }_1(s){\alpha }_{cs1}-{\mu }_2(s){\alpha }_{cs2}=0,\forall Att.$$

Full text

Introduction

Graph, as a kind of powerful data structure in non-Euclidean domain, can represent a set of instances (nodes) and the relationships (edges) between them, thus has a broad application in various fields (?). Different from regular Euclidean data such as texts, images and videos, which have clear grid structures that are relatively easy to generalize fundamental mathematical operations (?), graph structured data are irregular so it is not straightforward to apply important operations in deep learning (e.g. convolutions). Consequently, the analysis of graph-structured data remains a challenging and ubiquitous question.

In recent years, Graph Neural Networks (GNNs) have been proposed to learn the representations of graph-structured data and attract a growing interest (?; ?; ?; ?; ?; ?; ?; ?; ?; ?). GNNs can iteratively update node embeddings by aggregating/passing node features and structural information in the graph. The generated node embeddings can be fed into an extra classification/prediction layer and the whole model is trained end-to-end for different tasks.

Though many GNNs have been proposed, it is noted that when updating the embedding of a node ${v_{i}}$ by aggregating the embeddings of its neighbor nodes ${v_{j}}$, most of the GNN variants will assign non-parametric weight between ${v_{i}}$ and ${v_{j}}$ in their aggregators (?; ?; ?). However, such aggregators (e.g. sum or mean) fail to learn and distinguish the information between a target node and its neighbors during the training. Taking account of different contributions from the nodes in a graph is important in real-world data as not all edges have similar impacts. A natural alternative solution is making the edge weights trainable to have a better expressive capability.

To assign learnable weights in the aggregation, attention mechanism (?; ?) is incorporated in GNNs. Thus the weights can be directly represented by attention coefficients between nodes and give interpretability (?; ?; ?). Though GNNs with the attention-based aggregators achieve promising performance on various tasks empirically, a clear understanding of their discriminative power is missing for the designing of more powerful attention-based GNNs. Recent works (?; ?; ?) have theoretically analyzed the expressive power of GNNs. However, they are unaware of the attention mechanism in their analysis. So that it’s unclear whether using attention mechanism in aggregation will constrain the expressive power of GNNs.

In this work, we make efforts to theoretically analyze the discriminative power of GNNs with attention-based aggregators. Our findings reveal that previous proposed attention-based aggregators fail to distinguish certain distinct structures. By determining all such cases, we reveal the reason for those failures is the ignorance of cardinality information in aggregation. It inspires us to improve the attention mechanism via cardinality preservation. We propose models that can be applied to any kind of attention mechanisms and achieve the goal. In our experiments on node and graph classifications, we confirm our theoretical analysis and validate the power of our proposed models. The best-performed one can achieve competitive results comparing to other baselines. Specifically, our key contributions are summarized as follows:

• •

We show that previously proposed attention-based aggregators in message-passing GNNs always fail to distinguish certain distinct structures. We determine all of those cases and demonstrate the reason is the ignorance of the cardinality information in attention-based aggregation.

• •

We propose Cardinality Preserved Attention (CPA) methods to improve the original attention-based aggregator. With them, we can distinguish all cases that previously always fail an attention-based aggregator.

• •

Experiments on node and graph classification validate our theoretical analysis and the power of our CPA models. Comparing to baselines, CPA models can reach state-of-the-art level.

Preliminaries

Notations

Let $G=(V,E)$ be a graph with set of nodes $V$ and set of edges $E$. The nearest neighbors of node $i$ are defined as $\mathcal{N}(i)=\{j|d(i,j)=1\}$, where $d(i,j)$ is the shortest distance between node $i$ and $j$. We denote the set of node $i$ and its nearest neighbors as $\tilde{\mathcal{N}}(i)=\mathcal{N}(i)\cup\{i\}$. For the nodes in $\tilde{\mathcal{N}}(i)$, their feature vectors form a multiset $M(i)=(S_{i},\mu_{i})$, where $S_{i}=\left\{s_{1},\ldots,s_{n}\right\}$ is the ground set of $M(i)$, and $\mu_{i}:S_{i}\rightarrow\mathbb{N}^{*}$ is the multiplicity function that gives the multiplicity of each $s\in S_{i}$. The cardinality $|M|$ of a multiset is the number of elements (with multiplicity) in the multiset.

Graph Neural Networks

General GNNs

Graph Neural Networks (GNNs) adopt element (node or edge) features $X$ and the graph structure $A$ as input to learn the representation of each element, $h_{i}$, or each graph, $h_{G}$, for different tasks. In this work, we focus on the GNNs under massage-passing framework, which updates the node embeddings by aggregating its nearest neighbor node embeddings iteratively. In previous surveys, this type of GNNs is referred as Graph Convolutional Networks in (?) or the GNNs with convolutional aggregator in (?). Under the framework, a learned representation of the node after $l$ aggregation layers can contain the features and the structural information within $l$-step neighborhoods of the node. The $l$-th layer of a GNN can be formally represented as:

[TABLE]

where the superscript $l$ denotes the $l$-th layer and $h_{i}^{0}$ is initialized as $X_{i}$. The aggregation function $\phi$ in equation 1 propagates information between nodes and updates the hidden state of nodes.

In the final layer, since the node representation $h_{i}^{L}$ after $L$ iterations contains the $L$-step neighborhood information, it can be directly used for local/node-level tasks. While for global/graph-level tasks, the whole graph representation $h_{G}$ is needed, which requiring an extra readout function $g$ to compute $h_{G}$ from all $h_{i}^{L}$:

[TABLE]

Attention-Based GNNs

In a GNN, when the aggregation function $\phi$ in equation 1 adopts attention mechanism, we consider it as an attention-based GNN. In previous survey (Section 6 of (?)), this is referred to the first two types of attentions which have been applied to graph data. The attention-based aggregator in $l$-th layer can be formulated as follows:

[TABLE]

where the superscript $l$ denotes the $l$-th layer and $e_{ij}$ is the attention coefficient computed by an attention function $Att$ to measure the relation between node $i$ and node $j$. $\alpha_{ij}$ is the attention weight calculated by the softmax function. Equation 5 is a weighted summation that uses all $\alpha$ as weights followed with a nonlinear function $f$.

Related Works

Since GNNs have achieved remarkable results in practice, a clear understanding of the power of GNNs in graph representational learning is needed to design better models and make further improvements. Recent works (?; ?; ?) focus on understanding the discriminative power of GNNs by comparing it to the Weisfeiler-Lehman (WL) test (?) when deciding the graph isomorphism. It is proved that massage-passing-based GNNs which aggregate the nearest neighbor node features of a node for embedding are at most as powerful as the 1-WL test (?). Inspired by the higher discriminative power of the $k$-WL test ($k>2$) (?) than the 1-WL test, GNNs that have a theoretically higher discriminative power than the massage-passing-based GNNs have been proposed based on the $k$-WL test (?; ?). However, the GNNs proposed in those works don’t specifically contain the attention mechanism as the part of their analysis. So it’s currently unknown whether the attention mechanism will constrain the discriminative power. Our work focuses on the massage-passing-based GNNs with attention mechanism, which are upper bounded by the 1-WL test.

Another recent work (?) aims to understand the attention mechanism over nodes in GNNs with experiments in a controlled environment. However, the attention mechanism discussed in the work is used in the pooling layer for the pooling of nodes, while our work investigates the usage of attention mechanism in the aggregation layer for the updating of nodes.

Limitation of Attention-Based GNNs

In this section, we theoretically analyze the discriminative power of attention-based GNNs and show their limitations. The discriminative power means how well an attention-based GNN can distinguish different elements (local or global structures). We find that previously proposed attention-based GNNs can fail in certain cases and the discriminative power is limited. Besides, by theoretically finding out all cases that always fail an attention-based GNN, we reveal that those failures come from the lack of cardinality preservation in attention-based aggregators. The details of proofs are included in the Supplemental Material.

Discriminative Power of Attention-based GNNs

We assume the node input feature space is countable. For any attention-based GNNs, we give the conditions in Lemma 1 to make them reach the upper bound of discriminative power when distinguishing different elements (local or global structures). In particular, each local structure belongs to a node and is the $k$-height subtree structure rooted at the node, which is naturally captured in the node feature $h_{i}^{k}$ after $k$ iterations in a GNN. The global structure contains the information of all such subtrees in a graph.

Lemma 1**.**

Let $\mathcal{A}:\mathcal{G}\rightarrow\mathbb{R}^{g}$ be a GNN following the neighborhood aggregation scheme with the attention-based aggregator (Equation 5). For global-level task, an extra readout function (Equation 2) is used in the final layer. $\mathcal{A}$ can reach its upper bound of discriminative power (can distinguish all distinct local structures or be as powerful as the 1-WL test when distinguishing distinct global structures) after sufficient iterations with the following conditions:

• •

Local-level*: Function $f$ and the weighted summation in Equation 5 are injective.*

• •

Global-level*: Besides the conditions for local-level, $\mathcal{A}$’s readout function (Equation 2) is injective.*

With Lemma 1, we are interested in whether its conditions can always be satisfied, so as to reach the upper bound of discriminative capacity of an attention-based GNN. Since the function $f$ and the global-level readout function can be predetermined to be injective, we focus on whether the weighted summation function in attention-based aggregator can be injective.

The Non-Injectivity of Attention-Based Aggregator

In this part, we aim to answer the following two questions:

Q 1**.**

Can the attention-based GNNs actually reach the upper bound of discriminative power? In other words, can the weighted summation function in an attention-based aggregator be injective?

Q 2**.**

If not, can we determine all of the cases that prevent any kind of weighted summation function being injective?

Given a countable feature space $\mathcal{H}$, a weighted summation function is a mapping $W:\mathcal{H}\rightarrow\mathbb{R}^{n}$. The exact $W$ is determined by the attention weights $\alpha$ computed from $Att$ in Equation 3. Since $Att$ is affected by stochastic optimization algorithms (e.g. SGD) which introduce stochasticity in $W$, we have to pay attention that $W$ is not fixed when dealing with the two questions.

In Theorem 1, we answer Q1 with No by giving the cases that make $W$ not to be injective. So that the attention-based GNNs can never meet their upper bound of discriminative power, which is stated in Corollary 1. Moreover, we answer Q2 with Yes in Theorem 1 by pointing out those cases are the only reason to always prevent $W$ being injective. This alleviates the difficulty of summarizing the properties of those cases. Besides, we can specifically propose methods to avoid those cases so as to let $W$ to be injective.

Theorem 1**.**

Assume the input feature space $\mathcal{X}$ is countable. Given a multiset $X\subset\mathcal{X}$ and the node feature $c$ of the central node, the weighted summation function $h(c,X)$ in aggregation is defined as $h(c,X)=\sum\nolimits_{x\in X}\alpha_{cx}f(x)$, where $f:\mathcal{X}\rightarrow\mathbb{R}^{n}$ is a mapping of input feature vector and $\alpha_{cx}$ is the attention weight between $f(c)$ and $f(x)$ calculated by the attention function $Att$ in Equation 3 and the softmax function in Equation 4. For all $f$ and $Att$, $h(c_{1},X_{1})=h(c_{2},X_{2})$ if and only if $c_{1}=c_{2}$, $X_{1}=(S,\mu)$ and $X_{2}=(S,k\cdot\mu)$ for $k\in\mathbb{N}^{*}$. In other words, $h$ will map different multisets to the same embedding if and only if the multisets have the same central node feature and the same distribution of node features.

Corollary 1**.**

Let $\mathcal{A}$ be the GNN defined in Lemma 1. $\mathcal{A}$ never reaches its upper bound of discriminative power:

There exists two different subtrees $S_{1}$ and $S_{2}$ or two graphs $G_{1}$ and $G_{2}$ that the Weisfeiler-Lehman test decides as non-isomorphic, such that $\mathcal{A}$ always maps the two subtrees/graphs to the same embeddings.

Attention Mechanism Fails to Preserve Cardinality

With Theorem 1, we are now interested in the properties of all cases that always prevent the weighted summation functions $W$ being injective. Since the multisets that all $W$ fail to distinguish share the same distribution of node features, we can say that $W$ ignores the multiplicity information of each identical element in the multisets. Thus the cardinality of the multiset is not preserved:

Corollary 2**.**

Let $\mathcal{A}$ be the GNN defined in Lemma 1. The attention-based aggregator in $\mathcal{A}$ cannot preserve the cardinality information of the multiset of node features in aggregation.

In the next section, we aim to propose improved attention-based models to preserve the cardinality in aggregation.

Cardinality Preserved Attention (CPA) Model

Since the cardinality of the multiset is not preserved in attention-based aggregators, our goal is to propose modifications to any kind of attention mechanism to make them capture the cardinality information. So that all of the cases that always prevent attention-based aggregator being injective can be avoided.

To achieve our goal, we modify the weighted summation function in Equation 5 to incorporate the cardinality information and don’t change the attention function in Equation 3 so as to keep its original expressive power. Two different models named as Additive and Scaled are proposed to modify the Original model in Equation 5:

Model 1**.**

(Additive)

[TABLE]

Model 2**.**

(Scaled)

[TABLE]

where $w$ is a non-zero vector $\in\mathbb{R}^{n}$, $\odot$ denotes the element-wise multiplication, $|\tilde{\mathcal{N}}(i)|$ equals to the cardinality of the multiset $\tilde{\mathcal{N}}(i)$, $\psi:\mathbb{Z}^{+}\rightarrow\mathbb{R}^{n}$ is an injective function.

In the Additive model, each element in the multiset will contribute to the term that we added to preserve the cardinality information. In the Scaled model, the original weighted summation is directly multiplied by a representational vector of the cardinality value. So with these models, distinct multisets with the same distribution will result in different embedding $h$. Note that both of our models don’t change the $Att$ function, such that they can keep the learning power of the original attention mechanism. We summarize the effect of our models in Corollary 3 and illustrate it in Figure 1.

Corollary 3**.**

Let $\mathcal{T}$ be the original attention-based aggregator in Equation 5. With our proposed Cardinality Preserved Attention (CPA) models in Equation 6 and 7, $\mathcal{T}^{\prime}s$ discriminative power is increased: $\mathcal{T}$ can now distinguish all different multisets in aggregation that it previously always fails to distinguish.

While the original attention-based aggregator is never injective as we mentioned in previous sections, our cardinality preserved attention-based aggregator can be injective with certain learned attention weights to reach its upper bound of discriminative power. We validate this in our experiments.

For the time and space complexity of our CPA models comparing to the original attention-based aggregator, it is obvious that the Model 1 and 2 take more time and space than the original one due to our introduced vectors $w$ and $\psi(|\tilde{\mathcal{N}}(i)|)$. Thus we further simplify our models by fixing the values in $w$ and $\psi(|\tilde{\mathcal{N}}(i)|)$ and define two CPA variants:

Model 3**.**

(f-Additive)

[TABLE]

Model 4**.**

(f-Scaled)

[TABLE]

Model 3 and 4 still preserve the cardinality information and have reduced time and space complexity comparing to Model 1 and 2. Actually, since $w$ and $\psi(|\tilde{\mathcal{N}}(i)|)$ are degenerate into constants, Model 3 and 4 have the same time and space complexity as the original model in Equation 5. In our experiments, we will examine all 4 models together with the original one.

Experiments

In our experiments, we focus on the following questions:

Q 3**.**

Since attention-based GNNs (e.g. GAT) are originally proposed for local-level tasks like node classification, will those models fail or not meet the upper bound of discriminative power when solving certain node classification tasks? If so, can our proposed CPA models improve the original model?

Q 4**.**

For global-level tasks like graph classification, how well can the original attention-based GNNs perform? Can our proposed CPA models improve the original model?

Q 5**.**

How the attention-based GNNs with our CPA models perform compared to baselines?

To answer Question 3, we design a node classification task which is to predict whether or not a node is included in a triangle as one vertex in a graph. To answer Question 4 and 5, we perform experiments on graph classification benchmarks and evaluate the performance of attention-based GNNs with CPA models.

Experimental Setup

Datasets

In our synthetic task (TRIANGLE-NODE) for predicting whether or not a node is included in a triangle, we generate a graph with different node features. In our experiment on graph classification, we use 6 benchmark datasets: 2 social network datasets (REDDIT-BINARY (RE-B), REDDIT-MULTI5K (RE-M5K)) and 4 bioinformatics datasets (MUTAG, PROTEINS, ENZYMES, NCI1). More details of the datasets are provided in Supplemental Material.

Models

In our experiments, the Original model is the one that uses the original version of an attention mechanism. We apply each of our 4 CPA models (Additive, Scaled, f-Additive and f-Scaled) to the original attention mechanism for comparison. In the Additive and Scaled models, we take advantage of the approximation capability of multi-layer perceptron (MLP) (?; ?) to model $f$ and $\psi$.

For node classification, we use GAT (?) as the Original model. For graph classification, we build a GNN (GAT-GC) based on GAT as the Original model: We adopt the attention mechanism in GAT to specify the form of Equation 3: $e_{ij}=\operatorname{LeakyReLU}\left(\mathbf{a}^{\top}\left[\mathbf{W}{h}_{i}\|\mathbf{W}{h}_{j}\right]\right)$. For the readout function, a naive way is to only consider the node embeddings from the last iteration. Although a sufficient number of iterations can help to avoid the cases in Theorem 1 by aggregating more diverse node features, the features from the latter iterations may generalize worse and the GNNs usually have shallow structures (?; ?). So the GAT-GC adopts the same function as used in (?; ?; ?; ?), which concatenates graph embeddings from all iterations: h_{G}=\operatorname*{\scalebox{1.0}[1.0]{\parallel}}_{k=0}^{L}\big{(}\operatorname{Readout}(\left\{h_{i}^{l}\big{|}i\in G\right\})\big{)}, $\operatorname{Readout}$ function can be sum or mean. With CPA models, the cases in Theorem 1 can be avoided in each iteration. Full experimental settings are included in Supplemental Material.

Node Classification

For the TRIANGLE-NODE dataset, the proportion P of multisets that hold the properties in Theorem 1 is $29.2\%$, as shown in Table 1. The classification accuracy of the Original model (GAT) is significantly lower than the CPA models. It supports the claim in Corollary 1: the Original model fails to distinguish all distinct multisets in the dataset and exhibits constrained discriminate power. On the contrary, CPA models can distinguish all different multisets in the graph as suggested in Corollary 3 and indeed significantly improve the accuracy of the Original model as shown in Table 1. This experiment thus well answers Question 3 that we raised.

Graph Classification

In this section, we aim to answer Question 4 by evaluating the performance of variants of GAT-based GNN (GAT-GC) on graph classification benchmarks. Besides, we compare our best-performed CPA model with baseline models to answer Question 5.

Social Network Datasets

Since the RE-B and RE-M5K datasets don’t have original node features and we assign all the node features to be the same, we have $P=100.0\%$ in those datasets. Thus all multisets in aggregation will be mapped to the same embedding by the Original GAT-GC. After a mean readout function on all multisets, all graphs are finally mapped to the same embedding. The performance of the Original model is just random guessing of the graph labels as reported in Table 2. While our CPA models can distinguish all different multisets and are confirmed to be significantly better than the Original one.

Here we examine a naive approach to incorporate the cardinality information in the Original model by assigning node degrees as input node labels. By doing this way, the node features are diverse and we get $P=0.0\%$, which means that the cases in Theorem 1 can be all avoided. However, the testing accuracies of Original can only reach $76.65\pm 9.87\%$ on RE-B and $43.71\pm 9.05\%$ on RE-M5K, which are significantly lower than the results of CPA models in Table 2. Thus in practice, our proposed models exhibit good generalization power comparing to the naive approach.

Bioinformatics Datasets

For bioinformatics datasets that contain diverse node labels, we also report the $P$ values in Table 3. The results reveal the existence ($P\geq 29.3\%$) of the cases in those datasets that can fool the Original model, thus the discriminative power of the Original model is theoretically constrained.

To empirically validate this, we compare the training accuracies of GAT-GC variants, since the discriminative power can be directly indicated by the accuracies on training sets. Higher training accuracy indicates a better fitting ability to distinguish different graphs. The training curves of GAT-GC variants are shown in Figure 2. From these curves, we can see even though the Original model has overfitted different datasets, the fitting accuracies that it converges to can never be higher than those of our CPA models. Compared to the WL kernel, CPA models can get training accuracies close to $100\%$ on several datasets, which reach those obtained from the WL kernel (equal to $100\%$ as shown in (?)). These findings validate that the discriminative power of the Original model is constrained while our CPA models can approach the upper bound of discriminative power with certain learned weights.

In Table 3 we report the testing accuracies of GAT-GC variants on bioinformatics datasets. The Original model can get meaningful results. However, we find our proposed CPA models further improve the testing accuracies of the Original model on all datasets. This indicates that the preservation of cardinality can also benefit the generalization power of the model besides the discriminative power.

From previous results in Table 2 and 3, we find the f-Scaled model performs the best with an average ranking measure (?). The good performance of the fixed-weight models (f-Additive and f-Scaled) comparing to the full models (Additive and Scaled) demonstrates that the improvements achieved by CPA models are not simply due to the increased capacities given by the additional vectors embedded.

Comparison to Baselines

We further compare the best-performed GAT-GC variant (f-Scaled) with other baselines (WL kernel (WL) (?), PATCHY-SAN (PSCN) (?), Deep Graph CNN (DGCNN) (?), Graph Isomorphism Network (GIN) (?) and Capsule Graph Neural Network (CapsGNN) (?)). In Table 4, we report the results. Our GAT-GC (f-Scaled) model achieves 4 top 1 and 2 top 2 on all 6 datasets. It is expected that even better performance can be achieved with certain choices of attention mechanism besides the GAT one.

Conclusion

In this paper, we theoretically analyze the representational power of GNNs with attention-based aggregators: We determine all cases when those GNNs always fail to distinguish distinct structures. The finding shows that the missing cardinality information in aggregation is the only reason to cause those failures. To improve, we propose cardinality preserved attention (CPA) models to solve this issue. In our experiments, we validate our theoretical analysis that the performances of the original attention-based GNNs are limited. With our models, the original models can be improved. Compared to other baselines, our best-performed model achieves competitive performance. In future work, a challenging problem is to better learn the attention weights so as to guarantee the injectivity of our cardinality preserved attention models after the training. Besides, it would be interesting to analyze the effects of different attention mechanisms.

Appendix A Proof for Lemma 1

Proof.

Local-level: For the aggregator in the first layer, it will map different 1-height subtree structures to different embeddings from the distinct input multisets of neighborhood node features, since it’s injective. Iteratively, the aggregator in the $l$-th layer can distinguish different $l$-height subtree structures by mapping them to different embeddings from the distinct input multisets of $l$-1-height subtree features, since it’s injective.

Global level: From Lemma 2 and Theorem 3 in (?), we know: When all functions in $\mathcal{A}$ are injective, $\mathcal{A}$ can reach its upper bound of discriminative power, which is the same as the Weisfeiler-Lehman (WL) test (?) when deciding the graph isomorphism. ∎

Appendix B Proof for Theorem 1

Proof.

To prove Theorem 1, we have consider both two directions in the iff statement:

(1)

If given $c_{1}=c_{2}=c$, $X_{1}=(S,\mu)$ and $X_{2}=(S,k\cdot\mu)$, as $h(c,X)=\sum_{x\in X}\alpha_{cx}f(x)$, we have:

[TABLE]

where $\alpha_{cxi}$ is the attention weight belongs to $X_{i}$, and between $f(c)$ and $f(x)$, $x\in X_{i},i\in\{1,2\}$.

We can rewrite the equations using $S$ and $\mu$:

[TABLE]

[TABLE]

where $\mu(s)$ is the multiplicity function, and $\alpha_{csi}$ is the attention weight belongs to $X_{i}$, and between $f(c)$ and $f(s)$, $s\in S,i\in\{1,2\}$.

Considering the softmax function in Equation 2 of our paper, we can use attention coefficient $e$ to rewrite the equations:

[TABLE]

where $e_{csi}$ is the attention coefficient belongs to $X_{i}$, and between $f(c)$ and $f(s)$, $s\in S,i\in\{1,2\}$. Moreover, $e_{cxi}$ is the attention coefficient belongs to $X_{i}$, and between $f(c)$ and $f(x)$, $x\in X_{i},i\in\{1,2\}$.

As attention coefficient $e$ is computed by function $Att$, which is regardless of $X$, thus $e_{cs1}=e_{cs2}$, $\forall s\in S$ and $e_{cx1}=e_{cx2}$, $\forall x\in X_{1},X_{2}$. We denote $e_{cx}=e_{cx1}=e_{cx2}$, $e_{cs}=e_{cs1}=e_{cs2}$. Remind that $X_{2}$ has $k$ copies of the elements in $X_{1}$, so that

[TABLE]

Using this equation, we can get

[TABLE]

From all equations above, we finally have

[TABLE]

(2)

If given $h(c_{1},X_{1})=h(c_{2},X_{2})$ for all $f$, $Att$, we have

[TABLE]

where $\alpha_{cxi}$ is the attention weight belongs to $X_{i}$, and between $f(c_{i})$ and $f(x)$, $x\in X_{i},i\in\{1,2\}$.

We denote $X_{1}=(S_{1},\mu_{1})$ and $X_{2}=(S_{2},\mu_{2})$ and rewrite the equation:

[TABLE]

where $\mu_{i}(s)$ is the multiplicity function of $X_{i},i\in\{1,2\}$. Moreover, $\alpha_{csi}$ is the attention weight belongs to $X_{i}$, and between $f(c_{i})$ and $f(s)$, $s\in S_{i},i\in\{1,2\}$.

When considering the relations between $S_{1}$ and $S_{2}$, we have:

[TABLE]

If we assume the equality of Equation 10 is true for all $f$ and $S_{1}\neq S_{2}$, we can define such two functions $f_{1}$ and $f_{2}$:

[TABLE]

If given the equality of Equation 10 is true for $f_{1}$, we have:

[TABLE]

We can rewrite Equation 11 using $f_{2}$:

[TABLE]

Thus we know

[TABLE]

Note that the LHS of Equation 12 is just the LHS of Equation 10 when $f=f_{2}$. As $\mu_{i}(s)\geq 1$ due to the definition of multiplicity, $\alpha_{csi}>0$ due to the softmax function, we have $\mu_{i}(s)\alpha_{csi}>0,\forall s\in S_{i},i\in\{1,2\}$. Thus the RHS of Equation 12 > 0 and we now know the equality in Equation 10 is not true for $f_{2}$. So the assumption of $S_{1}\neq S_{2}$ is false.

We denote $S=S_{1}=S_{2}$. To let the remaining summation term always equal to 0, we have

[TABLE]

Considering Equation 2 in our paper, we can rewrite the equation above:

[TABLE]

where $e_{csi}$ is the attention coefficient belongs to $X_{i}$, and between $f(c_{i})$ and $f(s)$, $s\in S$. And $e_{cxi}$ is the attention coefficient belongs to $X_{i}$, and between $f(c_{i})$ and $f(x)$, $x\in X_{i},i\in\{1,2\}$.

The LHS of Equation 13 is a rational number. However if $c_{1}\neq c_{2}$, the RHS of Equation 13 can be irrational: We assume $S$ contains at least two elements $s_{0}$ and $s\neq s_{0}$. If not, we can directly get $c_{1}=c_{2}$. We consider any attention mechanism that results in:

[TABLE]

Thus when $s=s_{0}$, the RHS of the equation become:

[TABLE]

where $n$ is the multiplicity of $s_{0}$ in $X_{2}$. It is obvious that the value of RHS is irrational. So we have $c_{1}=c_{2}$ to always hold the equality.

With $c_{1}=c_{2}$, we know $e_{cs1}=e_{cs2}$, $\forall s\in S$ and $e_{cx1}=e_{cx2}$, $\forall x\in X_{1},X_{2}$. We denote $e_{cx}=e_{cx1}=e_{cx2}$, Equation 13 becomes

[TABLE]

We further denote $k=\mu_{1}(s)/\mu_{2}(s),\forall s\in S$. So that $\mu_{2}=k\cdot\mu_{1}$. Finally by denoting $\mu=\mu_{1}$, we have $X_{1}=(S,\mu)$, $X_{2}=(S,k\cdot\mu)$ and $c_{1}=c_{2}$. ∎

Appendix C Proof for Corollary 1

Proof.

For subtrees, if $S_{1}$ and $S_{2}$ are 1-height subtrees that have the same root node feature and the same distribution of node features, $\mathcal{A}$ will get the same embeddings for $S_{1}$ and $S_{2}$ according to Theorem 1.

For graphs, let $G_{1}$ be a fully connect graph with $n$ nodes and $G_{2}$ be a ring-like graph with $n$ nodes. All nodes in $G_{1}$ and $G_{2}$ have the same feature $x$. It is clear that the Weisfeiler-Lehman test of isomorphism decides $G_{1}$ and $G_{2}$ as non-isomorphic.

We denote $\{X_{i}\},i\in G_{1}$ as the set of multisets for aggregation in $G_{1}$, and $\{X_{j}\},j\in G_{2}$ as the set of multisets for aggregation in $G_{2}$. As $G_{1}$ is a fully connect graph, all multisets in $G_{1}$ contain $1$ central node and $n-1$ neighbors. As $G_{2}$ is a ring-like graph, all multisets in $G_{2}$ contain $1$ central node and $2$ neighbors. Thus we have

[TABLE]

[TABLE]

where $\mu_{i}(x)$ is the multiplicity function of the node with feature $x$ in $G_{i},i\in\{1,2\}$.

From Theorem 1, we know that $h(c,X_{i})=h(c,X_{j}),\forall i\in G_{1},\forall j\in G_{2}$. Considering the Equation 3 of our paper, we have $h_{i}^{l}=h_{j}^{l},\forall i\in G_{1},\forall j\in G_{2}$ in each iteration $l$. Besides, as the number of node in $G_{1}$ and $G_{2}$ are equals to $n$, $\mathcal{A}$ will always map $G_{1}$ and $G_{2}$ to the same set of multisets of node features $\{h^{l}\}$ in each iteration $l$ and finally get the same embedding for each graph. ∎

Appendix D Proof for Corollary 2

Proof.

Given two distinct multiset of node features $X_{1}$ and $X_{2}$ that have the same central node feature and the same distribution of node features: $c_{1}=c_{2}$, $X_{1}=(S,\mu)$ and $X_{2}=(S,k\cdot\mu)$ for $k\in\mathbb{N}^{*}$, we know the cardinality of $X_{2}$ is $k$ times of the cardinality of $X_{1}$. Thus $X_{1}$ and $X_{2}$ can be distinguished by their cardinality.

However, the weighted summation function $h$ in attention-based aggregator $\mathcal{A}$ will map them to the same embedding: $h(c_{1},X_{1})=h(c_{2},X_{2})$ according to Theorem 1. Thus we cannot distinguish $X_{1}$ and $X_{2}$ via $\mathcal{A}$. To conclude, $\mathcal{A}$ lost the cardinality information after aggregation. ∎

Appendix E Proof for Corollary 3

Proof.

For any two distinct multisets $X_{1}$ and $X_{2}$ that $\mathcal{T}$ previously always fail to distinguish according to Theorem 1, we denote $X_{1}=(S,\mu)$ and $X_{2}=(S,k\cdot\mu)\subset\mathcal{X}$ for some $k\in\mathbb{N}^{*}$ and $c\in S$. Thus $\sum\nolimits_{x\in X_{1}}\alpha_{cx1}f(x)=\sum\nolimits_{x\in X_{2}}\alpha_{cx2}f(x)$, where $\alpha_{cxi}$ is the attention weight belongs to $X_{i}$, and between $f(c)$ and $f(x)$, $x\in X_{i},i\in\{1,2\}$. We denote $H=\sum_{x\in X_{1}}\alpha_{cx1}f(x)=\sum_{x\in X_{2}}\alpha_{cx2}f(x)$. When applying CPA models, the aggregation functions in $\mathcal{T}$ can be rewritten as:

[TABLE]

We consider the following example: All elements in $w$ equal to 1. Function $\psi$ maps \big{|}X\big{|} to a n-dimensional vector which all elements in it equal to \big{|}X\big{|}. And $f(x)=N^{-Z(x)}$, where $Z:\mathcal{X}\rightarrow\mathbb{N}$ and N>\big{|}X\big{|}. So that the aggregation functions become:

[TABLE]

For $h_{1}$, we have $h_{1}(c,X_{1})-h_{1}(c,X_{2})=\sum\nolimits_{x\in X_{1}}f(x)-\sum\nolimits_{x\in X_{2}}f(x)$. According to Lemma 5 of (?), when $X_{1}\neq X_{2}$, $\sum\nolimits_{x\in X_{1}}f(x)\neq\sum\nolimits_{x\in X_{2}}f(x)$. So $h_{1}(c,X_{1})\neq h_{1}(c,X_{2})$.

For $h_{2}$, we have h_{2}(c,X_{1})-h_{2}(c,X_{2})=(\big{|}X_{1}\big{|}-\big{|}X_{2}\big{|})\cdot H. As $\alpha_{cx}>0$ due to the softmax function, and $f(x)>0$ in our example, we know $H>0$. Moreover as \big{|}X_{1}\big{|}-\big{|}X_{2}\big{|}\neq 0, we can get $h_{2}(c,X_{1})\neq h_{2}(c,X_{2})$. ∎

Appendix F Details of Datasets

For the node classification task, we generate a graph with 4800 nodes and 32400 edges. $40.58\%$ of the nodes are included in triangles as vertices while $59.42\%$ are not. There are 4000 nodes assigned with feature ’0’, 400 with feature ’1’ and 400 with feature ’2’. The label of each node for prediction is whether or not it’s included in a triangle.

For the graph classification task, detailed statistics of the bioinformatics and social network datasets are listed in Table 5. All of the datasets are available at https://ls11-www.cs.tu-dortmund.de/staff/morris/graphkerneldatasets.

In all datasets, if the original node features are provided, we use the one-hot encodings of them as input.

Appendix G Details of Experiment Settings

For all experiments, we perform 10-fold cross-validation and repeat the experiments 10 times for each dataset and each model. To get a final accuracy for each run, we select the epoch with the best cross-validation accuracy averaged over all 10 folds. The average accuracies and their standard deviations are reported based on the results across the folds in all runs.

In our Additive and Scaled models, all MLPs have 2 layers with ReLU activation.

In the GAT variants, we use 2 GNN layers and a hidden dimensionality of 32. The negative input slope of $\operatorname{LeakyReLU}$ in the $\operatorname{GAT}$ attention mechanism is 0.2. The number of heads in multi-head attention is 1.

In the GAT-GC variants, we use 4 GNN layers. For the $\operatorname{Readout}$ function in all models, we use sum for bioinformatics datasets and mean for social network datasets. We apply Batch normalization (?) after every hidden layers. The hidden dimensionality is set as 32 for bioinformatics datasets and 64 for social network datasets. The negative input slope of $\operatorname{LeakyReLU}$ in the $\operatorname{GAT}$ attention mechanism is 0.2. We use a single head in the multi-head attention in all models.

All models are trained using the Adam optimizer (?) and the learning rate is dropped by a factor of 0.5 every 400 epochs in the node classification task and every 50 epochs in the graph classification task. We use an initial learning rate of 0.01 for the TRIANGLE-NODE and bioinformatics datasets and 0.0025 for the social network datasets. For the GAT variants, we use a dropout ratio of 0 and a weight decay value of 0. For the GAT-GC variants on each dataset, the following hyper-parameters are tuned: (1) Batch size in $\{32,128\}$; (2) Dropout ratio in $\{0,0.5\}$ after dense layer; (3) $L_{2}$ regularization from [math] to $0.001$. On each dataset, we use the same hyper-parameter configurations in all model variants for a fair comparison.

For the results of the baselines for comparison, we use the results reported in their original works by default. If results are not available, we use the best testing results reported in (?; ?).