A natural upper bound to the accuracy of predicting protein stability changes upon mutations

TL;DR

This paper establishes a theoretical upper limit on the accuracy of predicting protein stability changes upon mutations, considering data noise and distribution, revealing that current methods are close to this bound and that dataset differences can mislead comparisons.

Contribution

It introduces a novel analytical framework to estimate the maximum achievable prediction accuracy for protein stability changes based on data intrinsic properties.

Findings

The upper bound for prediction correlation is approximately 0.7-0.8.

Current predictors are near the theoretical performance limit.

Dataset noise and distribution significantly influence prediction accuracy.

Abstract

Accurate prediction of protein stability changes upon single-site variations (DDG) is important for protein design, as well as our understanding of the mechanism of genetic diseases. The performance of high-throughput computational methods to this end is evaluated mostly based on the Pearson correlation coefficient between predicted and observed data, assuming that the upper bound would be 1 (perfect correlation). However, the performance of these predictors can be limited by the distribution and noise of the experimental data. Here we estimate, for the first time, a theoretical upper-bound to the DDG prediction performances imposed by the intrinsic structure of currently available DDG data. Given a set of measured DDG protein variations, the theoretically best predictor is estimated based on its similarity to another set of experimentally determined DDG values. We investigate the correlation between pairs of measured DDG variations, where one is used as a predictor for the other. We analytically derive an upper bound to the Pearson correlation as a function of the noise and distribution of the DDG data. We also evaluate the available datasets to highlight the effect of the noise in conjunction with DDG distribution. We conclude that the upper bound is a function of both uncertainty and spread of the DDG values, and that with current data the best performance should be between 0.7-0.8, depending on the dataset used; higher Pearson correlations might be indicative of overtraining. It also follows that comparisons of predictors using different datasets are inherently misleading.