Is Transfer Learning Necessary for Protein Landscape Prediction?

TL;DR

This paper demonstrates that simple supervised CNN models can compete with or outperform complex pretraining-based models in protein landscape prediction tasks, questioning the necessity of transfer learning in this domain.

Contribution

It shows that straightforward supervised CNNs can match or surpass the performance of pretraining-based models on protein landscape tasks, emphasizing the value of simple baselines.

Findings

CNN models compete with TAPE models on key tasks

Linear regression outperforms CNNs on fluorescence prediction

Simple models are effective and easy to train

Abstract

Recently, there has been great interest in learning how to best represent proteins, specifically with fixed-length embeddings. Deep learning has become a popular tool for protein representation learning as a model's hidden layers produce potentially useful vector embeddings. TAPE introduced a number of benchmark tasks and showed that semi-supervised learning, via pretraining language models on a large protein corpus, improved performance on downstream tasks. Two of the tasks (fluorescence prediction and stability prediction) involve learning fitness landscapes. In this paper, we show that CNN models trained solely using supervised learning both compete with and sometimes outperform the best models from TAPE that leverage expensive pretraining on large protein datasets. These CNN models are sufficiently simple and small that they can be trained using a Google Colab notebook. We also find for the fluorescence task that linear regression outperforms our models and the TAPE models. The benchmarking tasks proposed by TAPE are excellent measures of a model's ability to predict protein function and should be used going forward. However, we believe it is important to add baselines from simple models to put the performance of the semi-supervised models that have been reported so far into perspective.