MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Use

TL;DR

MCPVerse is a comprehensive real-world benchmark with over 550 tools and 140k tokens, designed to evaluate and improve large language models' ability to use external tools effectively in complex, time-sensitive tasks.

Contribution

This paper introduces MCPVerse, the first expansive, real-world benchmark with a large tool set and outcome-based evaluation for assessing agentic tool use in LLMs.

Findings

Most models' performance drops with larger tool sets.

Claude-4-Sonnet effectively leverages expanded exploration.

MCPVerse sets a new standard for real-world agentic tool use evaluation.

Abstract

Large Language Models (LLMs) are evolving from text generators into reasoning agents. This transition makes their ability to use external tools a critical capability. However, evaluating this skill presents a significant challenge. Existing benchmarks are often limited by their reliance on synthetic tools and severely constrained action spaces. To address these limitations, we introduce MCPVerse, an expansive, real-world benchmark for evaluating agentic tool use. MCPVerse integrates more than 550 real-world, executable tools to create an unprecedented action space exceeding 140k tokens, and employs outcome-based evaluation with real-time ground truth for time-sensitive tasks. We benchmarked the state-of-the-art LLMs across three modes (Oracle, Standard, and Max-Scale), revealing that while most models suffer performance degradation when confronted with larger tool sets, the agentic models, such as Claude-4-Sonnet, can effectively leverage expanded exploration spaces to improve accuracy. This finding not only exposes the limitations of state-of-the-art models in complex, real-world scenarios but also establishes MCPVerse as a critical benchmark for measuring and advancing agentic tool use capabilities.