VibeProteinBench: An Evaluation Benchmark for Language-interfaced Vibe Protein Design

TL;DR

VibeProteinBench is a comprehensive benchmark designed to evaluate large language models' ability to perform various aspects of vibe protein design, including recognition, engineering, and generation, under open-ended conditions.

Contribution

It introduces an integrated, multi-stage benchmark for assessing the broad capabilities of language models in vibe protein design, filling a gap in existing evaluation frameworks.

Findings

No current model excels across all three design stages.

The benchmark reveals significant challenges for generalist models in protein design.

In silico validation confirms biological plausibility of model outputs.

Abstract

Protein design aims to compose amino-acid sequences that fold into stable three-dimensional structures while satisfying targeted functional properties. The field is increasingly shifting toward vibe protein design, where a single model is expected to generate novel sequences, engineer existing proteins, and reason about protein characteristics through flexible natural-language constraints. Large language models (LLMs) have emerged as a leading paradigm in this space. However, existing evaluation benchmarks often limit their scope to a partial aspect of protein design, while others restrict design objectives to structured input schemas, lacking an integrated framework that evaluates the broad spectrum of protein design competence under open-ended intents. To this end, we present Vibe Protein design Benchmark (VibeProteinBench), a language-interfaced benchmark that probes generalist capabilities through three complementary stages mirroring a computational protein design workflow: recognition, engineering, and generation. Each stage is grounded in expert-curated mechanistic rationales and multi-faceted in silico validation, to computationally verify whether model outputs are biologically plausible. Evaluations across diverse general-purpose and domain-specialized LLMs reveal that no model achieves strong performance across all three stages, suggesting that generalist protein design remains a substantial open challenge for current LLMs.