Antibody Foundational Model : Ab-RoBERTa

TL;DR

This paper introduces Ab-RoBERTa, a specialized antibody language model based on RoBERTa, designed to support antibody research with improved efficiency and accessibility, leveraging large antibody datasets.

Contribution

The study develops and releases Ab-RoBERTa, the first publicly available antibody-specific RoBERTa-based language model, tailored for antibody sequence analysis tasks.

Findings

Ab-RoBERTa demonstrates effective performance in antibody-related tasks.

The model is smaller and more efficient than previous BERT-based models.

Public availability facilitates broader antibody research applications.

Abstract

With the growing prominence of antibody-based therapeutics, antibody engineering has gained increasing attention as a critical area of research and development. Recent progress in transformer-based protein large language models (LLMs) has demonstrated promising applications in protein sequence design and structural prediction. Moreover, the availability of large-scale antibody datasets such as the Observed Antibody Space (OAS) database has opened new avenues for the development of LLMs specialized for processing antibody sequences. Among these, RoBERTa has demonstrated improved performance relative to BERT, while maintaining a smaller parameter count (125M) compared to the BERT-based protein model, ProtBERT (420M). This reduced model size enables more efficient deployment in antibody-related applications. However, despite the numerous advantages of the RoBERTa architecture, antibody-specific foundational models built upon it have remained inaccessible to the research community. In this study, we introduce Ab-RoBERTa, a RoBERTa-based antibody-specific LLM, which is publicly available at https://huggingface.co/mogam-ai/Ab-RoBERTa. This resource is intended to support a wide range of antibody-related research applications including paratope prediction or humanness assessment.