Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

TL;DR

This paper compares transformer-based protein language models to natural language models, revealing domain-specific differences and introducing an early-exit technique that enhances both accuracy and efficiency in protein property prediction.

Contribution

It provides a comparative analysis of attention distributions in protein versus natural language models and adapts an early-exit method to improve protein task performance and efficiency.

Findings

Attention distributions differ significantly between protein and natural language models.

Early-exit technique improves prediction accuracy by up to 7.01 percentage points.

Efficiency increases by over 10% in protein property prediction tasks.

Abstract

Modern Protein Language Models (PLMs) apply transformer-based model architectures from natural language processing to biological sequences, predicting a variety of protein functions and properties. However, protein language has key differences from natural language, such as a rich functional space despite a vocabulary of only 20 amino acids. These differences motivate research into how transformer-based architectures operate differently in the protein domain and how we can better leverage PLMs to solve protein-related tasks. In this work, we begin by directly comparing how the distribution of information stored across layers of attention heads differs between the protein and natural language domain. Furthermore, we adapt a simple early-exit technique-originally used in the natural language domain to improve efficiency at the cost of performance-to achieve both increased accuracy and substantial efficiency gains in protein non-structural property prediction by allowing the model to automatically select protein representations from the intermediate layers of the PLMs for the specific task and protein at hand. We achieve performance gains ranging from 0.4 to 7.01 percentage points while simultaneously improving efficiency by over 10 percent across models and non-structural prediction tasks. Our work opens up an area of research directly comparing how language models change behavior when moved into the protein domain and advances language modeling in biological domains.