QMBench: A Research Level Benchmark for Quantum Materials Research

TL;DR

QMBench is a comprehensive benchmark designed to evaluate large language models' ability to perform quantum materials research tasks, aiming to accelerate AI development in this scientific domain.

Contribution

This paper introduces QMBench, the first standardized benchmark for assessing AI models' capabilities in quantum materials research across multiple scientific domains.

Findings

QMBench covers structural, electronic, thermodynamic, and computational aspects.

It provides a framework for evaluating AI's research-level understanding.

Designed to evolve with community input.

Abstract

We introduce QMBench, a comprehensive benchmark designed to evaluate the capability of large language model agents in quantum materials research. This specialized benchmark assesses the model's ability to apply condensed matter physics knowledge and computational techniques such as density functional theory to solve research problems in quantum materials science. QMBench encompasses different domains of the quantum material research, including structural properties, electronic properties, thermodynamic and other properties, symmetry principle and computational methodologies. By providing a standardized evaluation framework, QMBench aims to accelerate the development of an AI scientist capable of making creative contributions to quantum materials research. We expect QMBench to be developed and constantly improved by the research community.