Graph-tree Fusion Model with Bidirectional Information Propagation for Long Document Classification

TL;DR

This paper introduces a graph-tree fusion model with bidirectional information flow that enhances long document classification by capturing hierarchical and contextual dependencies across multiple levels.

Contribution

It proposes a novel hierarchical structure combining syntax trees and document graphs with bidirectional propagation, improving long document understanding beyond existing methods.

Findings

Effective in handling arbitrarily long documents

Outperforms existing methods on classification tasks

Enables comprehensive hierarchical content modeling

Abstract

Long document classification presents challenges in capturing both local and global dependencies due to their extensive content and complex structure. Existing methods often struggle with token limits and fail to adequately model hierarchical relationships within documents. To address these constraints, we propose a novel model leveraging a graph-tree structure. Our approach integrates syntax trees for sentence encodings and document graphs for document encodings, which capture fine-grained syntactic relationships and broader document contexts, respectively. We use Tree Transformers to generate sentence encodings, while a graph attention network models inter- and intra-sentence dependencies. During training, we implement bidirectional information propagation from word-to-sentence-to-document and vice versa, which enriches the contextual representation. Our proposed method enables a comprehensive understanding of content at all hierarchical levels and effectively handles arbitrarily long contexts without token limit constraints. Experimental results demonstrate the effectiveness of our approach in all types of long document classification tasks.