Knowledge Graph and Hypergraph Transformers with Repository-Attention and Journey-Based Role Transport

TL;DR

This paper introduces a novel transformer architecture that jointly models sentences and structured data like knowledge graphs and hypergraphs, using repository-attention and journey-based role transport for improved knowledge integration.

Contribution

It proposes a new dual-stream transformer architecture with role-based attention mechanisms that separately encode linguistic and structured knowledge, enabling explicit separation and effective alignment.

Findings

Effective joint training on sentences and structured data.

Explicit separation of language and knowledge representations.

Enhanced knowledge graph and hypergraph encoding capabilities.

Abstract

We present a concise architecture for joint training on sentences and structured data while keeping knowledge and language representations separable. The model treats knowledge graphs and hypergraphs as structured instances with role slots and encodes them into a key-value repository that a language transformer can attend over. Attention is conditioned by journey-based role transport, which unifies edge-labeled KG traversal, hyperedge traversal, and sentence structure. We outline a dual-stream architecture, hierarchical layer groups with instance-local, neighborhood, and global mixing attention, retrieval over a separate repository, and multi-task objectives spanning masked language modeling, link prediction, and role-consistency denoising. The result is an explicit, inspectable separation between linguistic context and structured knowledge, while still enabling tight alignment through cross-attention.