Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning

TL;DR

This paper assesses the reasoning capabilities of various large language models by comparing their performance on logical and abstract reasoning tasks against human benchmarks, highlighting their strengths and limitations.

Contribution

It provides a comparative analysis of multiple LLMs on reasoning tasks, revealing their current limitations in logical deduction compared to humans.

Findings

LLMs show significant gaps in logical reasoning skills.

Performance varies widely across different models.

Identifies specific reasoning areas where LLMs struggle.

Abstract

Evaluating reasoning ability in Large Language Models (LLMs) is important for advancing artificial intelligence, as it transcends mere linguistic task performance. It involves understanding whether these models truly understand information, perform inferences, and are able to draw conclusions in a logical and valid way. This study compare logical and abstract reasoning skills of several LLMs - including GPT, Claude, DeepSeek, Gemini, Grok, Llama, Mistral, Perplexity, and Sabi\'a - using a set of eight custom-designed reasoning questions. The LLM results are benchmarked against human performance on the same tasks, revealing significant differences and indicating areas where LLMs struggle with deduction.