Long-context Protein Language Modeling Using Bidirectional Mamba with Shared Projection Layers

TL;DR

This paper introduces LC-PLM, a novel protein language model architecture based on structured state-space models, capable of handling longer protein sequences and biological interaction graphs, outperforming Transformer-based models on various tasks.

Contribution

The paper presents LC-PLM, an alternative to Transformer models for protein language modeling, with improved length extrapolation and biological context integration using structured state-space models.

Findings

LC-PLM outperforms ESM-2 on downstream tasks by up to 30%.

LC-PLM demonstrates better length extrapolation capabilities.

Incorporating PPI graphs enhances protein structure and function prediction.

Abstract

Self-supervised training of language models (LMs) has seen great success for protein sequences in learning meaningful representations and for generative drug design. Most protein LMs are based on the Transformer architecture trained on individual proteins with short context lengths. Such protein LMs cannot extrapolate to longer proteins and protein complexes well. They also fail to account for the underlying biological mechanisms carried out by biomolecular interactions and dynamics i.e., proteins often interact with other proteins, molecules, and pathways in complex biological systems. In this work, we propose LC-PLM based on an alternative protein LM architecture, BiMamba-S, built upon selective structured state-space models, to learn high-quality universal protein representations at the amino acid token level using masked language modeling. We also introduce its graph-contextual variant, LC-PLM, which contextualizes protein-protein interaction (PPI) graphs for a second stage of training. LC-PLM demonstrates favorable neural scaling laws, better length extrapolation capability, and up to 30% and 16% improvements on protein downstream tasks compared to Transformer-based ESM-2 when trained with 100B and 1T tokens, respectively. LC-PLM-G further trained within the context of PPI graphs shows promising results on protein structure and function prediction tasks. Our study demonstrates the benefit of increasing the context size with computationally efficient LM architecture (e.g., structured state space models) in learning universal protein representations and incorporating molecular interaction contexts contained in biological graphs.