ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimation

TL;DR

ChemAU is a novel framework that improves chemical reasoning in large language models by adaptively estimating uncertainty and integrating chemical expertise, leading to more accurate and reliable chemical problem-solving.

Contribution

This work introduces ChemAU, a new adaptive uncertainty estimation method that enhances LLM reasoning in chemistry by identifying knowledge gaps and integrating domain-specific expertise.

Findings

Significant improvement in reasoning accuracy across chemistry datasets.

Enhanced uncertainty estimation precisely identifies reasoning errors.

Effective correction of flawed reasoning chains in chemical problems.

Abstract

Large Language Models (LLMs) are widely used across various scenarios due to their exceptional reasoning capabilities and natural language understanding. While LLMs demonstrate strong performance in tasks involving mathematics and coding, their effectiveness diminishes significantly when applied to chemistry-related problems. Chemistry problems typically involve long and complex reasoning steps, which contain specific terminology, including specialized symbol systems and complex nomenclature conventions. These characteristics often cause general LLMs to experience hallucinations during the reasoning process due to their lack of specific knowledge. However, existing methods are struggling to effectively leverage chemical expertise and formulas. Moreover, current uncertainty estimation methods, designed to mitigate potential reasoning errors, are unable to precisely identify specific steps or key knowledge. In this work, we propose a novel framework called ChemAU, which incorporates our adaptive uncertainty estimation method that applies different uncertainty values based on the position of reasoning steps within the whole reasoning chain. Leveraging this method, ChemAU identifies gaps in chemistry knowledge and precisely supplements chemical expertise with the specialized domain model, thereby correcting and updating the previously flawed reasoning chain. Our experiments with three popular LLMs across three chemistry datasets demonstrate that ChemAU significantly enhances both reasoning accuracy and uncertainty estimation.