Accurate and efficient protein embedding using multi-teacher distillation learning

TL;DR

This paper introduces a multi-teacher distillation learning method for protein embedding that significantly reduces computational costs while maintaining high accuracy, facilitating large-scale protein analysis.

Contribution

The paper presents a novel distillation approach leveraging multiple pre-trained models to create efficient, compact protein embeddings with minimal performance loss.

Findings

Reduces computational time by approximately 70%.

Maintains nearly the same accuracy as larger models.

Enables efficient large-scale protein analysis.

Abstract

Motivation: Protein embedding, which represents proteins as numerical vectors, is a crucial step in various learning-based protein annotation/classification problems, including gene ontology prediction, protein-protein interaction prediction, and protein structure prediction. However, existing protein embedding methods are often computationally expensive due to their large number of parameters, which can reach millions or even billions. The growing availability of large-scale protein datasets and the need for efficient analysis tools have created a pressing demand for efficient protein embedding methods. Results: We propose a novel protein embedding approach based on multi-teacher distillation learning, which leverages the knowledge of multiple pre-trained protein embedding models to learn a compact and informative representation of proteins. Our method achieves comparable performance to state-of-the-art methods while significantly reducing computational costs and resource requirements. Specifically, our approach reduces computational time by ~70\% and maintains almost the same accuracy as the original large models. This makes our method well-suited for large-scale protein analysis and enables the bioinformatics community to perform protein embedding tasks more efficiently.