BioBridge: Bridging Proteins and Language for Enhanced Biological Reasoning with LLMs

TL;DR

BioBridge is a novel framework that enhances large language models with protein domain knowledge through continual pretraining and cross-modal alignment, improving performance on biological and general tasks.

Contribution

It introduces a domain-adaptive continual pretraining method and a cross-modal alignment pipeline to integrate protein knowledge into LLMs, enabling versatile biological reasoning.

Findings

Achieves comparable performance to specialized PLMs on protein benchmarks

Performs on par with LLMs on general understanding tasks

Effectively mitigates catastrophic forgetting during domain adaptation

Abstract

Existing Protein Language Models (PLMs) often suffer from limited adaptability to multiple tasks and exhibit poor generalization across diverse biological contexts. In contrast, general-purpose Large Language Models (LLMs) lack the capability to interpret protein sequences and fall short in domain-specific knowledge, limiting their capacity for effective biosemantic reasoning. To combine the advantages of both, we propose BioBridge, a domain-adaptive continual pretraining framework for protein understanding. This framework employs Domain-Incremental Continual Pre-training (DICP) to infuse protein domain knowledge and general reasoning corpus into a LLM simultaneously, effectively mitigating catastrophic forgetting. Cross-modal alignment is achieved via a PLM-Projector-LLM pipeline, which maps protein sequence embeddings into the semantic space of the language model. Ultimately, an end-to-end optimization is adopted to uniformly support various tasks, including protein property prediction and knowledge question-answering. Our proposed BioBridge demonstrates performance comparable to that of mainstream PLMs on multiple protein benchmarks, such as EC and BindingDB. It also achieves results on par with LLMs on general understanding tasks like MMLU and RACE. This showcases its innovative advantage of combining domain-specific adaptability with general-purpose language competency.