# ImmUQBench: a benchmark on uncertainty quantification of protein immunogenicity prediction

## TL;DR

This paper introduces ImmUQBench, a benchmark for evaluating uncertainty quantification methods in predicting protein immunogenicity, aiming to improve the reliability of AI/ML models in therapeutic antigen design.

## Contribution

The novel contribution is the creation of ImmUQBench, a standardized benchmark for assessing UQ methods in immunogenicity prediction under data-scarce conditions.

## Key findings

- Different UQ strategies show varying abilities in capturing predictive uncertainty and maintaining robustness.
- ImmUQBench provides a standardized evaluation framework for model accuracy, calibration, and robustness.
- The benchmark helps identify the most trustworthy UQ methods for immunogenicity prediction.

## Abstract

Discovering antigen proteins, capable of eliciting desired immune responses, is of paramount importance in developing immunogenic therapeutics for combating various diseases, particularly autoimmune disorders, infectious diseases, as well as cancers. Despite recent advances in artificial intelligence (AI) and machine learning (ML), accurate and generalizable immunogenicity prediction remains challenging due to limited labeled data and model over-simplifications. Uncertainty quantification (UQ) approaches are commonly used to address the aforementioned challenges when applying AI/ML methods with limited training data, aiming to reduce the risk of catastrophic errors. This study aims to systematically evaluate the performance of UQ methods for antigen immunogenicity prediction and to establish a benchmark for assessing model reliability in data-scarce setting. We here present ImmUQBench, a comprehensive benchmark that compares several well-known UQ methods for antigen immunogenicity prediction tasks. The benchmark assesses models in terms of predictive accuracy, calibration, and robustness under both in and out of distribution settings, providing standardized evaluation framework. Our evaluation reveals that different UQ strategies exhibit varying capabilities in capturing predictive uncertainty and maintaining robustness. This work yields critical insights into the performance and reliability of various UQ methods when applied to immunogenicity data, helping to identify which methods offer the most trustworthy predictions. ImmUQBench provides a unified platform for assessing UQ approaches in immunogenicity prediction, facilitating the development of more trustworthy AI/ML models for therapeutic antigen design. By offering insights into the strengths and limitations of existing UQ methods, our work facilitates more effective and reliable immunogenic therapeutic discovery.