Do Protein Transformers Have Biological Intelligence?

TL;DR

This paper introduces a new Transformer architecture for protein function prediction, a novel dataset, and an explainability technique, demonstrating that small models can effectively capture biological patterns and intelligence.

Contribution

The work presents a new Protein Transformer model, a comprehensive protein dataset, and an explainability method to interpret biological intelligence in protein sequences.

Findings

Small SPT-Tiny model achieves 94.3% accuracy on AR dataset.

Models discover biologically meaningful sequence patterns.

The Protein-FN dataset is publicly released for future research.

Abstract

Deep neural networks, particularly Transformers, have been widely adopted for predicting the functional properties of proteins. In this work, we focus on exploring whether Protein Transformers can capture biological intelligence among protein sequences. To achieve our goal, we first introduce a protein function dataset, namely Protein-FN, providing over 9000 protein data with meaningful labels. Second, we devise a new Transformer architecture, namely Sequence Protein Transformers (SPT), for computationally efficient protein function predictions. Third, we develop a novel Explainable Artificial Intelligence (XAI) technique called Sequence Score, which can efficiently interpret the decision-making processes of protein models, thereby overcoming the difficulty of deciphering biological intelligence bided in Protein Transformers. Remarkably, even our smallest SPT-Tiny model, which contains only 5.4M parameters, demonstrates impressive predictive accuracy, achieving 94.3% on the Antibiotic Resistance (AR) dataset and 99.6% on the Protein-FN dataset, all accomplished by training from scratch. Besides, our Sequence Score technique helps reveal that our SPT models can discover several meaningful patterns underlying the sequence structures of protein data, with these patterns aligning closely with the domain knowledge in the biology community. We have officially released our Protein-FN dataset on Hugging Face Datasets https://huggingface.co/datasets/Protein-FN/Protein-FN. Our code is available at https://github.com/fudong03/BioIntelligence.