Socialformer: Social Network Inspired Long Document Modeling for Document Ranking

TL;DR

Socialformer introduces social network-inspired sparse attention patterns in Transformer models to improve long document modeling for ranking tasks, addressing the limitations of local-only attention mechanisms.

Contribution

The paper proposes a novel attention pattern based on social network characteristics, enabling better remote connections in long document modeling within Transformers.

Findings

Effective long document modeling demonstrated on ranking tasks.

Sparse attention patterns improve computational efficiency.

Model outperforms existing methods in long document understanding.

Abstract

Utilizing pre-trained language models has achieved great success for neural document ranking. Limited by the computational and memory requirements, long document modeling becomes a critical issue. Recent works propose to modify the full attention matrix in Transformer by designing sparse attention patterns. However, most of them only focus on local connections of terms within a fixed-size window. How to build suitable remote connections between terms to better model document representation remains underexplored. In this paper, we propose the model Socialformer, which introduces the characteristics of social networks into designing sparse attention patterns for long document modeling in document ranking. Specifically, we consider several attention patterns to construct a graph like social networks. Endowed with the characteristic of social networks, most pairs of nodes in such a graph can reach with a short path while ensuring the sparsity. To facilitate efficient calculation, we segment the graph into multiple subgraphs to simulate friend circles in social scenarios. Experimental results confirm the effectiveness of our model on long document modeling.