Hierarchical Graph Topic Modeling with Topic Tree-based Transformer

TL;DR

This paper introduces a novel Transformer-based model that integrates hierarchical topic modeling within documents and across document graphs using hyperbolic space, enhancing semantic and structural understanding.

Contribution

It proposes a hierarchical graph topic modeling transformer that combines intra-document topic hierarchies and inter-document graph structures in hyperbolic space.

Findings

Effective in capturing hierarchical topic structures within documents.

Improves representation learning by integrating graph and topic hierarchies.

Validated through both supervised and unsupervised experiments.

Abstract

Textual documents are commonly connected in a hierarchical graph structure where a central document links to others with an exponentially growing connectivity. Though Hyperbolic Graph Neural Networks (HGNNs) excel at capturing such graph hierarchy, they cannot model the rich textual semantics within documents. Moreover, text contents in documents usually discuss topics of different specificity. Hierarchical Topic Models (HTMs) discover such latent topic hierarchy within text corpora. However, most of them focus on the textual content within documents, and ignore the graph adjacency across interlinked documents. We thus propose a Hierarchical Graph Topic Modeling Transformer to integrate both topic hierarchy within documents and graph hierarchy across documents into a unified Transformer. Specifically, to incorporate topic hierarchy within documents, we design a topic tree and infer a hierarchical tree embedding for hierarchical topic modeling. To preserve both topic and graph hierarchies, we design our model in hyperbolic space and propose Hyperbolic Doubly Recurrent Neural Network, which models ancestral and fraternal tree structure. Both hierarchies are inserted into each Transformer layer to learn unified representations. Both supervised and unsupervised experiments verify the effectiveness of our model.