Question Rephrasing for Quantifying Uncertainty in Large Language Models: Applications in Molecular Chemistry Tasks

TL;DR

This paper introduces a question rephrasing method combined with sampling techniques to better quantify the uncertainty of large language models in molecular chemistry applications, enhancing reliability assessment.

Contribution

The paper presents a novel question rephrasing approach integrated with sampling to evaluate input and output uncertainty in LLMs for chemistry tasks, which is a new methodology.

Findings

Improved uncertainty quantification in molecular property prediction.

Enhanced assessment of LLM reliability in chemical reaction prediction.

Validated approach on real-world chemistry datasets.

Abstract

Uncertainty quantification enables users to assess the reliability of responses generated by large language models (LLMs). We present a novel Question Rephrasing technique to evaluate the input uncertainty of LLMs, which refers to the uncertainty arising from equivalent variations of the inputs provided to LLMs. This technique is integrated with sampling methods that measure the output uncertainty of LLMs, thereby offering a more comprehensive uncertainty assessment. We validated our approach on property prediction and reaction prediction for molecular chemistry tasks.