A Syllogistic Probe: Tracing the Evolution of Logic Reasoning in Large Language Models

TL;DR

This paper investigates how large language models evolve in their logical reasoning capabilities, specifically their shift from traditional to modern logic, using syllogistic reasoning as a probe.

Contribution

It introduces a new syllogism dataset and demonstrates how model size, thinking processes, and base model influence the evolution of logical reasoning in LLMs.

Findings

Model size scaling promotes the shift toward modern logic

Thinking accelerates the logical evolution beyond parameter scaling

Base model significantly affects the stability of logical shift

Abstract

Human logic has gradually shifted from intuition-driven inference to rigorous formal systems. Motivated by recent advances in large language models (LLMs), we explore whether LLMs exhibit a similar evolution in the underlying logical framework. Using existential import as a probe, we for evaluate syllogism under traditional and modern logic. Through extensive experiments of testing SOTA LLMs on a new syllogism dataset, we have some interesting findings: (i) Model size scaling promotes the shift toward modern logic; (ii) Thinking serves as an efficient accelerator beyond parameter scaling; (iii) the Base model plays a crucial role in determining how easily and stably this shift can emerge. Beyond these core factors, we conduct additional experiments for in-depth analysis of properties of current LLMs on syllogistic reasoning.