PoET: A generative model of protein families as sequences-of-sequences

TL;DR

PoET is a novel autoregressive transformer model that generates and scores protein sequences within families, leveraging sequences-of-sequences modeling to improve transfer learning, extrapolation, and variant prediction.

Contribution

PoET introduces a new transformer architecture that models protein families as sequences-of-sequences, enabling better transfer learning and sequence generation across diverse protein families.

Findings

Outperforms existing models in variant function prediction

Effective on small and large protein families

Capable of controllable protein sequence generation

Abstract

Generative protein language models are a natural way to design new proteins with desired functions. However, current models are either difficult to direct to produce a protein from a specific family of interest, or must be trained on a large multiple sequence alignment (MSA) from the specific family of interest, making them unable to benefit from transfer learning across families. To address this, we propose $\textbf{P}$r$\textbf{o}$tein $\textbf{E}$volutionary $\textbf{T}$ransformer (PoET), an autoregressive generative model of whole protein families that learns to generate sets of related proteins as sequences-of-sequences across tens of millions of natural protein sequence clusters. PoET can be used as a retrieval-augmented language model to generate and score arbitrary modifications conditioned on any protein family of interest, and can extrapolate from short context lengths to generalize well even for small families. This is enabled by a unique Transformer layer; we model tokens sequentially within sequences while attending between sequences order invariantly, allowing PoET to scale to context lengths beyond those used during training. In extensive experiments on deep mutational scanning datasets, we show that PoET outperforms existing protein language models and evolutionary sequence models for variant function prediction across proteins of all MSA depths. We also demonstrate PoET's ability to controllably generate new protein sequences.