Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations

TL;DR

This paper introduces ChemCoTBench, a framework for evaluating large language models' chemical reasoning through modular, step-by-step operations that mimic mathematical proofs, aiming to improve AI's role in chemical discovery.

Contribution

It proposes a novel reasoning framework that formalizes chemical problem-solving into transparent workflows using modular operations, bridging the gap between abstract reasoning and practical chemical tasks.

Findings

Models show improved reasoning on molecular optimization tasks.

ChemCoTBench provides structured datasets and evaluation metrics.

Baseline evaluations demonstrate the framework's effectiveness.

Abstract

While large language models (LLMs) with Chain-of-Thought (CoT) reasoning excel in mathematics and coding, their potential for systematic reasoning in chemistry, a domain demanding rigorous structural analysis for real-world tasks like drug design and reaction engineering, remains untapped. Current benchmarks focus on simple knowledge retrieval, neglecting step-by-step reasoning required for complex tasks such as molecular optimization and reaction prediction. To address this, we introduce ChemCoTBench, a reasoning framework that bridges molecular structure understanding with arithmetic-inspired operations, including addition, deletion, and substitution, to formalize chemical problem-solving into transparent, step-by-step workflows. By treating molecular transformations as modular "chemical operations", the framework enables slow-thinking reasoning, mirroring the logic of mathematical proofs while grounding solutions in real-world chemical constraints. We evaluate models on two high-impact tasks: Molecular Property Optimization and Chemical Reaction Prediction. These tasks mirror real-world challenges while providing structured evaluability. By providing annotated datasets, a reasoning taxonomy, and baseline evaluations, ChemCoTBench bridges the gap between abstract reasoning methods and practical chemical discovery, establishing a foundation for advancing LLMs as tools for AI-driven scientific innovation.