# MolErr2Fix: Benchmarking LLM Trustworthiness in Chemistry via Modular Error Detection, Localization, Explanation, and Revision

## TL;DR

MolErr2Fix is a new benchmark designed to evaluate and improve the ability of large language models to accurately detect, explain, and correct errors in molecular descriptions, emphasizing chemical understanding and reasoning.

## Contribution

It introduces MolErr2Fix, a comprehensive benchmark with fine-grained error annotations for assessing LLMs' chemical error detection and correction capabilities.

## Key findings

- Current LLMs show significant performance gaps in chemical error correction.
- MolErr2Fix provides detailed annotations to facilitate targeted improvements.
- Benchmark supports future research in developing more reliable chemical LLMs.

## Abstract

Large Language Models (LLMs) have shown growing potential in molecular sciences, but they often produce chemically inaccurate descriptions and struggle to recognize or justify potential errors. This raises important concerns about their robustness and reliability in scientific applications. To support more rigorous evaluation of LLMs in chemical reasoning, we present the MolErr2Fix benchmark, designed to assess LLMs on error detection and correction in molecular descriptions. Unlike existing benchmarks focused on molecule-to-text generation or property prediction, MolErr2Fix emphasizes fine-grained chemical understanding. It tasks LLMs with identifying, localizing, explaining, and revising potential structural and semantic errors in molecular descriptions. Specifically, MolErr2Fix consists of 1,193 fine-grained annotated error instances. Each instance contains quadruple annotations, i.e,. (error type, span location, the explanation, and the correction). These tasks are intended to reflect the types of reasoning and verification required in real-world chemical communication. Evaluations of current state-of-the-art LLMs reveal notable performance gaps, underscoring the need for more robust chemical reasoning capabilities. MolErr2Fix provides a focused benchmark for evaluating such capabilities and aims to support progress toward more reliable and chemically informed language models. All annotations and an accompanying evaluation API will be publicly released to facilitate future research.