Understanding Syllogistic Reasoning in LLMs from Formal and Natural Language Perspectives

TL;DR

This paper investigates how large language models perform syllogistic reasoning from both formal logical and natural language perspectives, revealing varied capabilities and raising questions about their reasoning nature.

Contribution

It provides a comprehensive analysis of syllogistic reasoning in LLMs, comparing symbolic inference and natural language understanding across 14 models.

Findings

Some models achieve perfect symbolic reasoning

LLMs show inconsistent reasoning capabilities

Evidence suggests a shift towards formal reasoning mechanisms

Abstract

We study syllogistic reasoning in LLMs from the logical and natural language perspectives. In process, we explore fundamental reasoning capabilities of the LLMs and the direction this research is moving forward. To aid in our studies, we use 14 large language models and investigate their syllogistic reasoning capabilities in terms of symbolic inferences as well as natural language understanding. Even though this reasoning mechanism is not a uniform emergent property across LLMs, the perfect symbolic performances in certain models make us wonder whether LLMs are becoming more and more formal reasoning mechanisms, rather than making explicit the nuances of human reasoning.