On Memorization of Large Language Models in Logical Reasoning

TL;DR

This paper investigates whether large language models rely on memorization or genuine reasoning in solving logical puzzles, revealing that they develop reasoning skills alongside memorization, with implications for understanding their capabilities.

Contribution

It provides a systematic, quantitative analysis of memorization versus reasoning in LLMs on logical puzzles, combining experiments, perturbation tests, and internal probing.

Findings

LLMs memorize training puzzles with high accuracy after fine-tuning.

Fine-tuning improves generalization despite heavy memorization.

Models switch between reasoning and memorization depending on the puzzle variation.

Abstract

Large language models (LLMs) achieve good performance on challenging reasoning benchmarks, yet could also make basic reasoning mistakes. This contrasting behavior is puzzling when it comes to understanding the mechanisms behind LLMs' reasoning capabilities. One hypothesis is that the increasingly high and nearly saturated performance on common reasoning benchmarks could be due to the memorization of similar problems. In this paper, we systematically investigate this hypothesis with a quantitative measurement of memorization in reasoning tasks, using a dynamically generated logical reasoning benchmark based on Knights and Knaves (K&K) puzzles. We find that LLMs could interpolate and memorize the training puzzles (achieving near-perfect accuracy) after fine-tuning, yet they struggle with slight variations of these puzzles. On the other hand, we show that while fine-tuning leads to heavy memorization, it also consistently improves generalization performance. Through in-depth analyses with perturbation tests, cross difficulty-level transferability, probing model internals, and fine-tuning with wrong answers, we establish that LLMs develop reasoning skills on K&K puzzles alongside memorization. Finally, our analysis based on a per-sample memorization score sheds light on how LLMs switch between reasoning and memorization when solving logical puzzles. Our code and data are available at https://memkklogic.github.io.