MolRecBench-Wild: A Real-World Benchmark for Optical Chemical Structure Recognition

TL;DR

This paper introduces MolRecBench-Wild, a comprehensive real-world benchmark for Optical Chemical Structure Recognition, along with a new difficulty framework and a semantic representation language, revealing significant challenges for existing models.

Contribution

It presents MolRecBench-Wild, a large-scale real-world benchmark with a novel difficulty framework and a semantic language, advancing evaluation of OCSR systems in practical scenarios.

Findings

Models perform poorly on MolRecBench-Wild, indicating a large gap with real-world applications.

The MOSAIC difficulty framework effectively characterizes challenges in molecular diagram recognition.

The CARBON language enables more faithful semantic representation of chemical structures.

Abstract

Optical Chemical Structure Recognition (OCSR) aims to translate molecular diagrams in scientific literature into machine-readable formats, but current systems remain unreliable on real-world images due to substantial visual and chemical complexity. We introduce MOSAIC, a dual-dimensional difficulty framework with 37 fine-grained labels that jointly characterize visual interference and chemical semantic challenges in molecular diagrams. Based on this framework, we construct MolRecBench-Wild, a benchmark of 5,029 structures from 820 recent chemistry papers, covering the full difficulty spectrum observed in real publications. To enable faithful semantic evaluation beyond SMILES and MolFile, we propose CARBON, a representation language capable of expressing valence variations, icon-based groups, and other non-standard chemical semantics. We further adopt a dual-track evaluation protocol supporting both CARBON and SMILES outputs for broad model compatibility. Comprehensive experiments over 18 OCSR-capable models reveal severe performance degradation on MolRecBench-Wild, exposing a large gap between previous patent benchmarks and real-world academic scenarios.