ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domain

TL;DR

ChemSafetyBench is a comprehensive benchmark designed to evaluate the safety and accuracy of large language models in chemistry, addressing critical vulnerabilities and promoting safer AI development in scientific research.

Contribution

Introduces ChemSafetyBench, a new dataset and evaluation framework for assessing LLM safety and accuracy in chemistry-related tasks, with over 30,000 samples and diverse scenarios.

Findings

State-of-the-art LLMs show strengths in chemical queries.

Critical safety vulnerabilities are identified in current models.

Benchmark promotes development of safer chemistry AI tools.

Abstract

The advancement and extensive application of large language models (LLMs) have been remarkable, including their use in scientific research assistance. However, these models often generate scientifically incorrect or unsafe responses, and in some cases, they may encourage users to engage in dangerous behavior. To address this issue in the field of chemistry, we introduce ChemSafetyBench, a benchmark designed to evaluate the accuracy and safety of LLM responses. ChemSafetyBench encompasses three key tasks: querying chemical properties, assessing the legality of chemical uses, and describing synthesis methods, each requiring increasingly deeper chemical knowledge. Our dataset has more than 30K samples across various chemical materials. We incorporate handcrafted templates and advanced jailbreaking scenarios to enhance task diversity. Our automated evaluation framework thoroughly assesses the safety, accuracy, and appropriateness of LLM responses. Extensive experiments with state-of-the-art LLMs reveal notable strengths and critical vulnerabilities, underscoring the need for robust safety measures. ChemSafetyBench aims to be a pivotal tool in developing safer AI technologies in chemistry. Our code and dataset are available at https://github.com/HaochenZhao/SafeAgent4Chem. Warning: this paper contains discussions on the synthesis of controlled chemicals using AI models.