DuConTE: Dual-Granularity Text Encoder with Topology-Constrained Attention for Text-attributed Graphs

TL;DR

DuConTE is a dual-granularity text encoder with topology-aware attention that improves text-attributed graph processing by integrating structural dependencies into semantic encoding.

Contribution

It introduces a cascaded architecture of two pretrained language models with topology-constrained attention for enhanced graph node representation.

Findings

Achieves state-of-the-art results on multiple benchmark datasets.

Effectively captures structural dependencies in text-attributed graphs.

Improves semantic encoding by considering node and neighborhood contexts.

Abstract

Text-attributed graphs integrate semantic information of node texts with topological structure, offering significant value in various applications such as document classification and information extraction. Existing approaches typically encode textual content using language models (LMs), followed by graph neural networks (GNNs) to process structural information. However, during the LM-based text encoding phase, most methods not only perform semantic interaction solely at the word-token granularity, but also neglect the structural dependencies among texts from different nodes. In this work, we propose DuConTE, a dual-granularity text encoder with topology-constrained attention. The model employs a cascaded architecture of two pretrained LMs, encoding semantics first at the word-token granularity and then at the node granularity. During the self-attention computation in each LM, we dynamically adjust the attention mask matrix based on node connectivity, guiding the model to learn semantic correlations informed by the graph structure. Furthermore, when composing node representations from word-token embeddings, we separately evaluate the importance of tokens under the center-node context and the neighborhood context, enabling the capture of more contextually relevant semantic information. Extensive experiments on multiple benchmark datasets demonstrate that DuConTE achieves state-of-the-art performance on the majority of them.