ReLSO: A Transformer-based Model for Latent Space Optimization and Generation of Proteins

TL;DR

ReLSO is a transformer-based autoencoder that models protein sequence-function landscapes, enabling efficient generation and optimization of high-fitness protein sequences through a structured latent space and gradient-based methods.

Contribution

This paper introduces ReLSO, a novel deep transformer autoencoder with a structured latent space for simultaneous sequence generation and fitness prediction in proteins.

Findings

ReLSO outperforms other methods in sequence optimization efficiency.

ReLSO more robustly generates high-fitness protein sequences.

Attention mechanisms in ReLSO provide insights into sequence-function relationships.

Abstract

The development of powerful natural language models have increased the ability to learn meaningful representations of protein sequences. In addition, advances in high-throughput mutagenesis, directed evolution, and next-generation sequencing have allowed for the accumulation of large amounts of labeled fitness data. Leveraging these two trends, we introduce Regularized Latent Space Optimization (ReLSO), a deep transformer-based autoencoder which features a highly structured latent space that is trained to jointly generate sequences as well as predict fitness. Through regularized prediction heads, ReLSO introduces a powerful protein sequence encoder and novel approach for efficient fitness landscape traversal. Using ReLSO, we explicitly model the sequence-function landscape of large labeled datasets and generate new molecules by optimizing within the latent space using gradient-based methods. We evaluate this approach on several publicly-available protein datasets, including variant sets of anti-ranibizumab and GFP. We observe a greater sequence optimization efficiency (increase in fitness per optimization step) by ReLSO compared to other approaches, where ReLSO more robustly generates high-fitness sequences. Furthermore, the attention-based relationships learned by the jointly-trained ReLSO models provides a potential avenue towards sequence-level fitness attribution information.