Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling

TL;DR

Ankh is a cost-effective, protein-specific language model that outperforms larger models in understanding protein structure and function, emphasizing accessibility and data efficiency.

Contribution

We introduce Ankh, a novel general-purpose protein language model trained efficiently with fewer resources, achieving state-of-the-art results and insights into protein evolution and diversity.

Findings

Ankh surpasses state-of-the-art performance with fewer parameters.

It effectively learns protein conservation-mutation trends.

It maintains structural and functional integrity in generated variants.

Abstract

As opposed to scaling-up protein language models (PLMs), we seek improving performance via protein-specific optimization. Although the proportionality between the language model size and the richness of its learned representations is validated, we prioritize accessibility and pursue a path of data-efficient, cost-reduced, and knowledge-guided optimization. Through over twenty experiments ranging from masking, architecture, and pre-training data, we derive insights from protein-specific experimentation into building a model that interprets the language of life, optimally. We present Ankh, the first general-purpose PLM trained on Google's TPU-v4 surpassing the state-of-the-art performance with fewer parameters (<10% for pre-training, <7% for inference, and <30% for the embedding dimension). We provide a representative range of structure and function benchmarks where Ankh excels. We further provide a protein variant generation analysis on High-N and One-N input data scales where Ankh succeeds in learning protein evolutionary conservation-mutation trends and introducing functional diversity while retaining key structural-functional characteristics. We dedicate our work to promoting accessibility to research innovation via attainable resources.