Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models

TL;DR

Multi-LogiEval introduces a comprehensive dataset for evaluating large language models' multi-step logical reasoning across various logic types and inference rules, revealing significant performance drops with increased reasoning depth.

Contribution

The paper presents Multi-LogiEval, a new dataset for multi-step logical reasoning evaluation, including non-monotonic logic, and provides extensive analysis of LLMs' reasoning capabilities.

Findings

LLMs' accuracy drops from ~68% at depth-1 to ~43% at depth-5.

Performance varies significantly across different logic types and inference rules.

Zero-shot chain-of-thought prompting reveals limitations in current LLM reasoning abilities.

Abstract

As Large Language Models (LLMs) continue to exhibit remarkable performance in natural language understanding tasks, there is a crucial need to measure their ability for human-like multi-step logical reasoning. Existing logical reasoning evaluation benchmarks often focus primarily on simplistic single-step or multi-step reasoning with a limited set of inference rules. Furthermore, the lack of datasets for evaluating non-monotonic reasoning represents a crucial gap since it aligns more closely with human-like reasoning. To address these limitations, we propose Multi-LogiEval, a comprehensive evaluation dataset encompassing multi-step logical reasoning with various inference rules and depths. Multi-LogiEval covers three logic types--propositional, first-order, and non-monotonic--consisting of more than 30 inference rules and more than 60 of their combinations with various depths. Leveraging this dataset, we conduct evaluations on a range of LLMs including GPT-4, ChatGPT, Gemini-Pro, Yi, Orca, and Mistral, employing a zero-shot chain-of-thought. Experimental results show that there is a significant drop in the performance of LLMs as the reasoning steps/depth increases (average accuracy of ~68% at depth-1 to ~43% at depth-5). We further conduct a thorough investigation of reasoning chains generated by LLMs which reveals several important findings. We believe that Multi-LogiEval facilitates future research for evaluating and enhancing the logical reasoning ability of LLMs. Data is available at https://github.com/Mihir3009/Multi-LogiEval.