Exploiting LLMs for Automatic Hypothesis Assessment via a Logit-Based Calibrated Prior

TL;DR

This paper introduces a novel LLM-based prior for automatically assessing the novelty of correlations in data, aiding hypothesis evaluation by predicting correlation values with high accuracy and generalization.

Contribution

It proposes the Logit-based Calibrated Prior, a new method leveraging LLMs to predict correlation values, outperforming existing classifiers and demonstrating context-sensitive reasoning.

Findings

Achieves 78.8% sign accuracy in correlation prediction

Outperforms fine-tuned RoBERTa in binary correlation prediction

Generalizes to unseen correlations, indicating reasoning beyond memorization

Abstract

As hypothesis generation becomes increasingly automated, a new bottleneck has emerged: hypothesis assessment. Modern systems can surface thousands of statistical relationships-correlations, trends, causal links-but offer little guidance on which ones are novel, non-trivial, or worthy of expert attention. In this work, we study the complementary problem to hypothesis generation: automatic hypothesis assessment. Specifically, we ask: given a large set of statistical relationships, can we automatically assess which ones are novel and worth further exploration? We focus on correlations as they are a common entry point in exploratory data analysis that often serve as the basis for forming deeper scientific or causal hypotheses. To support automatic assessment, we propose to leverage the vast knowledge encoded in LLMs' weights to derive a prior distribution over the correlation value of a variable pair. If an LLM's prior expects the correlation value observed, then such correlation is not surprising, and vice versa. We propose the Logit-based Calibrated Prior, an LLM-elicited correlation prior that transforms the model's raw output logits into a calibrated, continuous predictive distribution over correlation values. We evaluate the prior on a benchmark of 2,096 real-world variable pairs and it achieves a sign accuracy of 78.8%, a mean absolute error of 0.26, and 95% credible interval coverage of 89.2% in predicting Pearson correlation coefficient. It also outperforms a fine-tuned RoBERTa classifier in binary correlation prediction and achieves higher precision@K in hypothesis ranking. We further show that the prior generalizes to correlations not seen during LLM pretraining, reflecting context-sensitive reasoning rather than memorization.