Absurd World: A Simple Yet Powerful Method to Absurdify the Real-world for Probing LLM Reasoning Capabilities

TL;DR

Absurd World is a benchmarking framework that tests LLM reasoning by creating absurd scenarios with preserved logic, revealing their ability to think logically beyond learned real-world patterns.

Contribution

The paper introduces Absurd World, a novel benchmarking method for evaluating LLM reasoning robustness using logically coherent absurd scenarios.

Findings

LLMs struggle with absurd scenarios despite logical coherence.

Prompting techniques influence LLM performance on absurd tasks.

Absurd World effectively isolates reasoning ability from real-world pattern learning.

Abstract

While extremely powerful and versatile at various tasks, the thinking capabilities of large language models (LLMs) are often put under scrutiny as they sometimes fail to solve problems that humans can systematically solve. However, recent literature focuses on breaking LLM reasoning with increasingly complex problems, and whether an LLM is robust in simple logical reasoning remains underexplored. This paper proposes Absurd World, a benchmarking framework, to test LLMs against altered realism, where scenarios are logically coherent, and humans can easily solve the tasks. Absurd World breaks a real-world model into symbols, actions, sequences, and events, which are automatically altered to create absurd worlds where the logic to solve the tasks remains the same. It evaluates a large collection of models with simple and advanced prompting techniques, and proves that it is an effective tool to determine LLMs' ability to think logically, ignoring the patterns learned from the real world. One can use this framework to extensively test an LLM against a real-world problem to verify whether the LLM's reasoning capability is robust against variations of the task.