PDFBench: A Benchmark for De novo Protein Design from Function

TL;DR

PDFBench introduces a comprehensive evaluation framework for function-guided de novo protein design, enabling fair comparison of models across multiple metrics and settings, thus advancing research in drug discovery and enzyme engineering.

Contribution

The paper presents the first unified benchmark for protein design evaluation, systematically assessing multiple models and metrics to improve comparability and understanding.

Findings

Benchmarking eight models across 16 metrics reveals diverse strengths and weaknesses.

Correlation analysis provides insights into metric relationships and evaluation robustness.

SwissTest dataset ensures data integrity with a strict datetime cutoff.

Abstract

Function-guided protein design is a crucial task with significant applications in drug discovery and enzyme engineering. However, the field lacks a unified and comprehensive evaluation framework. Current models are assessed using inconsistent and limited subsets of metrics, which prevents fair comparison and a clear understanding of the relationships between different evaluation criteria. To address this gap, we introduce PDFBench, the first comprehensive benchmark for function-guided denovo protein design. Our benchmark systematically evaluates eight state-of-the-art models on 16 metrics across two key settings: description-guided design, for which we repurpose the Mol-Instructions dataset, originally lacking quantitative benchmarking, and keyword-guided design, for which we introduce a new test set, SwissTest, created with a strict datetime cutoff to ensure data integrity. By benchmarking across a wide array of metrics and analyzing their correlations, PDFBench enables more reliable model comparisons and provides key insights to guide future research.