THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Models

TL;DR

This paper introduces Think-Bench, a benchmark for assessing the reasoning efficiency of large reasoning models, highlighting prevalent overthinking issues and proposing metrics to improve computational resource utilization.

Contribution

It presents a systematic benchmark and novel metrics to evaluate and analyze the reasoning efficiency and chain-of-thought quality of large reasoning models.

Findings

Most LRMs overthink easy questions, producing unnecessarily long reasoning chains.

Many LRMs have high chain-of-thought quality but low efficiency.

Overthinking leads to significant computational resource waste.

Abstract

Large reasoning models (LRMs) have achieved impressive performance in complex tasks, often outperforming conventional large language models (LLMs). However, the prevalent issue of overthinking severely limits their computational efficiency. Overthinking occurs when models generate excessive and redundant tokens that contribute little to accurate outcomes, especially in simple tasks, resulting in a significant waste of computational resources. To systematically investigate this issue, we introduce Think-Bench, a benchmark designed to evaluate the reasoning efficiency of LRMs. We also propose novel efficiency metrics and conduct a comprehensive evaluation of various LRMs across multiple dimensions, including the reasoning process, outcome quality, and chain-of-thought (CoT) characteristics. Our analysis reveals that most LRMs exhibit overthinking in handling easy questions, generating unnecessarily lengthy reasoning chains. While many LRMs demonstrate high CoT quality, several suffer from low efficiency. We hope that Think-Bench can serve as a robust foundation for advancing research into LRMs.