PHAGE: Patent Heterogeneous Attention-Guided Graph Encoder for Representation Learning

TL;DR

PHAGE introduces a novel graph-based patent claim encoder that leverages hierarchical claim dependencies and heterogeneous relations to improve representation learning for patent analysis.

Contribution

It proposes a deterministic graph construction and token-level attention mechanism that incorporate claim hierarchy and relation types into a Transformer-based encoder.

Findings

PHAGE outperforms baseline models on classification, retrieval, and clustering tasks.

Intra-document claim topology is a stronger inductive bias than inter-document structure.

Claim topology representations persist in encoder weights after training.

Abstract

Patent claims form a directed dependency structure in which dependent claims inherit and refine the scope of earlier claims; however, existing patent encoders linearize claims as text and discard this hierarchy. Directly encoding this structure into self-attention poses two challenges: claim dependencies mix relation types that differ in semantics and extraction reliability, and the dependency graph is defined over claims while Transformers attend over tokens. PHAGE addresses the first challenge through a deterministic graph construction pipeline that separates near-deterministic legal citations from noisier rule-based technical relations, preserving type distinctions as heterogeneous edges. It addresses the second through a connectivity mask and learnable relation-aware biases that lift claim-level topology into token-level attention, allowing the encoder to differentially weight each relation type. A dual-granularity contrastive objective then aligns representations with both inter-patent taxonomy and intra-patent topology. PHAGE outperforms all baselines on classification, retrieval, and clustering, showing that intra-document claim topology is a stronger inductive bias than inter-document structure and that this bias persists in the encoder weights after training.