SUPERChem: A Multimodal Reasoning Benchmark in Chemistry

TL;DR

SUPERChem is a comprehensive, multimodal benchmark with expert-curated chemistry problems designed to evaluate and advance the reasoning capabilities of large language models beyond simple accuracy metrics.

Contribution

It introduces a new challenging benchmark with process-level evaluation and a scoring method for reasoning quality, addressing limitations of previous chemistry reasoning assessments.

Findings

GPT-5 achieves 38.5% accuracy, close to human baseline.

Multimodal information influences model reasoning differently.

The benchmark distinguishes high-fidelity reasoners from heuristic approaches.

Abstract

Current benchmarks for evaluating the chemical reasoning capabilities of Large Language Models (LLMs) are limited by oversimplified tasks, lack of process-level evaluation, and misalignment with expert-level chemistry skills. To address these issues, we introduce SUPERChem, a benchmark of 500 expert-curated reasoning-intensive chemistry problems, covering diverse subfields and provided in both multimodal and text-only formats. Original content and an iterative curation pipeline eliminate flawed items and mitigate data contamination. Each problem is paired with an expert-authored solution path, enabling Reasoning Path Fidelity (RPF) scoring to evaluate reasoning quality beyond final-answer accuracy. Evaluations against a human baseline of 40.3% accuracy show that even the best-performing model, GPT-5 (High), reaches only 38.5%, followed closely by Gemini 2.5 Pro (37.9%) and DeepSeek-V3.1-Think (37.3%). SUPERChem elicits multi-step, multimodal reasoning, reveals model-dependent effects of visual information, and distinguishes high-fidelity reasoners from heuristic ones. By providing a challenging benchmark and a reliable evaluation framework, SUPERChem aims to facilitate the advancement of LLMs toward expert-level chemical intelligence. The dataset of the benchmark is available at https://huggingface.co/datasets/ZehuaZhao/SUPERChem.