Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning

TL;DR

This paper critically examines the methodological issues in evaluating the reasoning abilities of base large language models, emphasizing the mismatch between their training objectives and the assessment of reasoning quality.

Contribution

It highlights fundamental methodological pitfalls in current evaluation practices of base LLMs' reasoning, urging a re-evaluation of existing conclusions and future research directions.

Findings

Base LLMs generate conclusions as linguistic pattern coincidences.

Current evaluation methods may misjudge LLM reasoning abilities.

Assumptions about base LLMs' reasoning do not generalize to fine-tuned models.

Abstract

Existing work investigates the reasoning capabilities of large language models (LLMs) to uncover their limitations, human-like biases and underlying processes. Such studies include evaluations of base LLMs (pre-trained on unlabeled corpora only) for this purpose. Our position paper argues that evaluating base LLMs' reasoning capabilities raises inherent methodological concerns that are overlooked in such existing studies. We highlight the fundamental mismatch between base LLMs' pretraining objective and normative qualities, such as correctness, by which reasoning is assessed. In particular, we show how base LLMs generate logically valid or invalid conclusions as coincidental byproducts of conforming to purely linguistic patterns of statistical plausibility. This fundamental mismatch challenges the assumptions that (a) base LLMs' outputs can be assessed as their bona fide attempts at correct answers or conclusions; and (b) conclusions about base LLMs' reasoning can generalize to post-trained LLMs optimized for successful instruction-following. We call for a critical re-examination of existing work that relies implicitly on these assumptions, and for future work to account for these methodological pitfalls.