Counting Clues: A Lightweight Probabilistic Baseline Can Match an LLM

TL;DR

This paper introduces a simple probabilistic method, FBPR, that matches LLM performance on clinical diagnosis tasks, highlighting the value of explicit probabilistic baselines and their complementary strengths.

Contribution

The paper presents FBPR, a lightweight Naive Bayes-based approach that achieves comparable results to large language models using co-occurrence statistics from pretraining corpora.

Findings

FBPR matches LLM performance on MedQA diagnosis tasks.

FBPR and LLMs correctly answer different questions, showing complementary strengths.

Explicit probabilistic methods still account for significant performance in benchmark tasks.

Abstract

Large language models (LLMs) excel on multiple-choice clinical diagnosis benchmarks, yet it is unclear how much of this performance reflects underlying probabilistic reasoning. We study this through questions from MedQA, where the task is to select the most likely diagnosis. We introduce the Frequency-Based Probabilistic Ranker (FBPR), a lightweight method that scores options with a smoothed Naive Bayes over concept-diagnosis co-occurrence statistics from a large corpus. When co-occurrence statistics were sourced from the pretraining corpora for OLMo and Llama, FBPR achieves comparable performance to the corresponding LLMs pretrained on that same corpus. Direct LLM inference and FBPR largely get different questions correct, with an overlap only slightly above random chance, indicating complementary strengths of each method. These findings highlight the continued value of explicit probabilistic baselines: they provide a meaningful performance reference point and a complementary signal for potential hybridization. While the performance of LLMs seems to be driven by a mechanism other than simple frequency aggregation, we show that an approach similar to the historically grounded, low-complexity expert systems still accounts for a substantial portion of benchmark performance.