Procrustean Bed for AI-Driven Retrosynthesis: A Unified Framework for Reproducible Evaluation

TL;DR

This paper introduces RetroCast, a comprehensive evaluation framework for computer-aided synthesis planning that standardizes model outputs, enabling rigorous comparison and revealing insights into the performance and limitations of current algorithms.

Contribution

The paper presents RetroCast, a unified benchmarking suite with standardized evaluation metrics and platforms, facilitating reproducible and transparent assessment of synthesis planning algorithms.

Findings

High solvability scores often mask chemical invalidity.

A 'complexity cliff' shows search-based methods struggle with long-range plans.

Sequence-based algorithms outperform in reconstructing complex synthetic routes.

Abstract

Progress in computer-aided synthesis planning (CASP) is obscured by the lack of standardized evaluation infrastructure and the reliance on metrics that prioritize topological completion over chemical validity. We introduce RetroCast, a unified evaluation suite that standardizes heterogeneous model outputs into a common schema to enable statistically rigorous, apples-to-apples comparison. The framework includes a reproducible benchmarking pipeline with stratified sampling and bootstrapped confidence intervals, accompanied by SynthArena, an interactive platform for qualitative route inspection. We utilize this infrastructure to evaluate leading search-based and sequence-based algorithms on a new suite of standardized benchmarks. Our analysis reveals a divergence between "solvability" (stock-termination rate) and route quality; high solvability scores often mask chemical invalidity or fail to correlate with the reproduction of experimental ground truths. Furthermore, we identify a "complexity cliff" in which search-based methods, despite high solvability rates, exhibit a sharp performance decay in reconstructing long-range synthetic plans compared to sequence-based approaches. We release the full framework, benchmark definitions, and a standardized database of model predictions to support transparent and reproducible development in the field.