Do Protein Transformers Have Biological Intelligence?
Fudong Lin, Wanrou Du, Jinchan Liu, Tarikul Milon, Shelby Meche, Wu Xu, Xiaoqi Qin, and Xu Yuan

TL;DR
This paper introduces a new Transformer architecture for protein function prediction, a novel dataset, and an explainability technique, demonstrating that small models can effectively capture biological patterns and intelligence.
Contribution
The work presents a new Protein Transformer model, a comprehensive protein dataset, and an explainability method to interpret biological intelligence in protein sequences.
Findings
Small SPT-Tiny model achieves 94.3% accuracy on AR dataset.
Models discover biologically meaningful sequence patterns.
The Protein-FN dataset is publicly released for future research.
Abstract
Deep neural networks, particularly Transformers, have been widely adopted for predicting the functional properties of proteins. In this work, we focus on exploring whether Protein Transformers can capture biological intelligence among protein sequences. To achieve our goal, we first introduce a protein function dataset, namely Protein-FN, providing over 9000 protein data with meaningful labels. Second, we devise a new Transformer architecture, namely Sequence Protein Transformers (SPT), for computationally efficient protein function predictions. Third, we develop a novel Explainable Artificial Intelligence (XAI) technique called Sequence Score, which can efficiently interpret the decision-making processes of protein models, thereby overcoming the difficulty of deciphering biological intelligence bided in Protein Transformers. Remarkably, even our smallest SPT-Tiny model, which contains…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning in Bioinformatics · Bioinformatics and Genomic Networks · Cell Image Analysis Techniques
MethodsAbsolute Position Encodings · Layer Normalization · Byte Pair Encoding · Label Smoothing · Softmax · Dropout · Dense Connections · Transformer · Focus
