Always Tell Me The Odds: Fine-grained Conditional Probability Estimation

TL;DR

This paper introduces a new model for precise, fine-grained probability estimation conditioned on context, addressing the limitations of large language models in uncertain scenarios.

Contribution

It presents a novel approach combining data creation, scaling, and supervision to significantly improve probabilistic predictions of LLMs.

Findings

Our model outperforms existing methods by a large margin.

Systematic evaluations show improved calibration and accuracy.

The approach is effective across various conditional probability tasks.

Abstract

We present a state-of-the-art model for fine-grained probability estimation of propositions conditioned on context. Recent advances in large language models (LLMs) have significantly enhanced their reasoning capabilities, particularly on well-defined tasks with complete information. However, LLMs continue to struggle with making accurate and well-calibrated probabilistic predictions under uncertainty or partial information. While incorporating uncertainty into model predictions often boosts performance, obtaining reliable estimates of that uncertainty remains understudied. In particular, LLM probability estimates tend to be coarse and biased towards more frequent numbers. Through a combination of human and synthetic data creation and assessment, scaling to larger models, and better supervision, we propose a set of strong and precise probability estimation models. We conduct systematic evaluations across tasks that rely on conditional probability estimation and show that our approach consistently outperforms existing fine-tuned and prompting-based methods by a large margin.