A Comprehensive Evaluation of LLM Reasoning: From Single-Model to Multi-Agent Paradigms

TL;DR

This paper provides a comprehensive evaluation of various reasoning paradigms for Large Language Models, comparing their performance, cost-accuracy trade-offs, and introducing a new benchmark for semantic reasoning capabilities.

Contribution

It offers a unified analysis of single-model, chain-of-thought, and multi-agent reasoning paradigms, and introduces MIMeBench for assessing semantic abstraction and contrastive discrimination.

Findings

Increased structural complexity does not always improve reasoning performance.

Multi-agent systems can offer favorable cost-accuracy trade-offs.

Semantic capabilities are crucial for comprehensive reasoning assessment.

Abstract

Large Language Models (LLMs) are increasingly deployed as reasoning systems, where reasoning paradigms - such as Chain-of-Thought (CoT) and multi-agent systems (MAS) - play a critical role, yet their relative effectiveness and cost-accuracy trade-offs remain poorly understood. In this work, we conduct a comprehensive and unified evaluation of reasoning paradigms, spanning direct single-model generation, CoT-augmented single-model reasoning, and representative MAS workflows, characterizing their reasoning performance across a diverse suite of closed-form benchmarks. Beyond overall performance, we probe role-specific capability demands in MAS using targeted role isolation analyses, and analyze cost-accuracy trade-offs to identify which MAS workflows offer a favorable balance between cost and accuracy, and which incur prohibitive overhead for marginal gains. We further introduce MIMeBench, a new open-ended benchmark that targets two foundational yet underexplored semantic capabilities - semantic abstraction and contrastive discrimination - thereby providing an alternative evaluation axis beyond closed-form accuracy and enabling fine-grained assessment of semantic competence that is difficult to capture with existing benchmarks. Our results show that increased structural complexity does not consistently lead to improved reasoning performance, with its benefits being highly dependent on the properties and suitability of the reasoning paradigm itself. The codes are released at https://gitcode.com/HIT1920/OpenLLMBench.