Generative Capacity of Probabilistic Protein Sequence Models

TL;DR

This paper evaluates the ability of different probabilistic protein sequence models to reproduce complex mutation patterns, highlighting the superior generative capacity of Potts models over VAEs and site-independent models.

Contribution

The study introduces a new set of sequence statistics to quantitatively compare GPSMs and assesses their ability to replicate natural sequence covariation patterns.

Findings

Potts models best reproduce higher-order mutational statistics.

VAEs have intermediate generative capacity.

Site-independent models perform the least accurately.

Abstract

Potts models and variational autoencoders (VAEs) have recently gained popularity as generative protein sequence models (GPSMs) to explore fitness landscapes and predict the effect of mutations. Despite encouraging results, quantitative characterization and comparison of GPSM-generated probability distributions is still lacking. It is currently unclear whether GPSMs can faithfully reproduce the complex multi-residue mutation patterns observed in natural sequences arising due to epistasis. We develop a set of sequence statistics to assess the "generative capacity" of three GPSMs of recent interest: the pairwise Potts Hamiltonian, the VAE, and the site-independent model, using natural and synthetic datasets. We show that the generative capacity of the Potts Hamiltonian model is the largest, in that the higher order mutational statistics generated by the model agree with those observed for natural sequences. In contrast, we show that the VAE's generative capacity lies between the pairwise Potts and site-independent models. Importantly, our work measures GPSM generative capacity in terms of higher-order sequence covariation statistics which we have developed, and provides a new framework for evaluating and interpreting GPSM accuracy that emphasizes the role of epistasis.