QUITE: Quantifying Uncertainty in Natural Language Text in Bayesian Reasoning Scenarios

TL;DR

QUITE is a new dataset for Bayesian reasoning in natural language, featuring complex relationships and real-world scenarios, enabling better evaluation of models' reasoning abilities.

Contribution

The paper introduces QUITE, a comprehensive dataset with natural language premises and questions for Bayesian reasoning, addressing limitations of previous datasets.

Findings

Logic-based models outperform large language models in reasoning tasks.

Neuro-symbolic models show promise for complex reasoning.

QUITE enables more realistic evaluation of reasoning models.

Abstract

Reasoning is key to many decision making processes. It requires consolidating a set of rule-like premises that are often associated with degrees of uncertainty and observations to draw conclusions. In this work, we address both the case where premises are specified as numeric probabilistic rules and situations in which humans state their estimates using words expressing degrees of certainty. Existing probabilistic reasoning datasets simplify the task, e.g., by requiring the model to only rank textual alternatives, by including only binary random variables, or by making use of a limited set of templates that result in less varied text. In this work, we present QUITE, a question answering dataset of real-world Bayesian reasoning scenarios with categorical random variables and complex relationships. QUITE provides high-quality natural language verbalizations of premises together with evidence statements and expects the answer to a question in the form of an estimated probability. We conduct an extensive set of experiments, finding that logic-based models outperform out-of-the-box large language models on all reasoning types (causal, evidential, and explaining-away). Our results provide evidence that neuro-symbolic models are a promising direction for improving complex reasoning. We release QUITE and code for training and experiments on Github.