Reasoning over Uncertain Text by Generative Large Language Models

TL;DR

This paper introduces BLInD, a new dataset for testing probabilistic reasoning in LLMs, and evaluates prompting strategies that improve their reasoning over uncertain text.

Contribution

The paper presents BLInD, a specialized dataset for probabilistic reasoning, and proposes prompting methods that enhance LLMs' reasoning capabilities over uncertain information.

Findings

Prompting strategies improve LLM performance on probabilistic reasoning tasks.

BLInD reveals significant limitations of current LLMs in handling uncertainty.

Methods are effective across multiple LLM architectures.

Abstract

This paper considers the challenges Large Language Models (LLMs) face when reasoning over text that includes information involving uncertainty explicitly quantified via probability values. This type of reasoning is relevant to a variety of contexts ranging from everyday conversations to medical decision-making. Despite improvements in the mathematical reasoning capabilities of LLMs, they still exhibit significant difficulties when it comes to probabilistic reasoning. To deal with this problem, we introduce the Bayesian Linguistic Inference Dataset (BLInD), a new dataset specifically designed to test the probabilistic reasoning capabilities of LLMs. We use BLInD to find out the limitations of LLMs for tasks involving probabilistic reasoning. In addition, we present several prompting strategies that map the problem to different formal representations, including Python code, probabilistic algorithms, and probabilistic logical programming. We conclude by providing an evaluation of our methods on BLInD and an adaptation of a causal reasoning question-answering dataset. Our empirical results highlight the effectiveness of our proposed strategies for multiple LLMs.