Higher-Order Modular Attention: Fusing Pairwise and Triadic Interactions for Protein Sequences

TL;DR

HOMA introduces a unified attention mechanism combining pairwise and triadic interactions, improving protein sequence predictions by capturing complex dependencies more effectively than standard methods.

Contribution

The paper proposes Higher-Order Modular Attention (HOMA), a novel attention operator that explicitly incorporates triadic interactions for better modeling of protein sequences.

Findings

HOMA outperforms standard self-attention on three protein benchmarks.

Explicit triadic interactions enhance representational capacity.

HOMA achieves these improvements with manageable computational costs.

Abstract

Transformer self-attention computes pairwise token interactions, yet protein sequence to phenotype relationships often involve cooperative dependencies among three or more residues that dot product attention does not capture explicitly. We introduce Higher-Order Modular Attention, HOMA, a unified attention operator that fuses pairwise attention with an explicit triadic interaction pathway. To make triadic attention practical on long sequences, HOMA employs block-structured, windowed triadic attention. We evaluate on three TAPE benchmarks for Secondary Structure, Fluorescence, and Stability. Our attention mechanism yields consistent improvements across all tasks compared with standard self-attention and efficient variants including block-wise attention and Linformer. These results suggest that explicit triadic terms provide complementary representational capacity for protein sequence prediction at controllable additional computational cost.