Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios

TL;DR

UltraTool is a new benchmark that assesses large language models' ability to plan, create, and use tools effectively in complex, real-world scenarios, addressing limitations of previous simple query-based evaluations.

Contribution

The paper introduces UltraTool, a comprehensive benchmark that evaluates LLMs' tool utilization in realistic, complex tasks with independent planning and no pre-defined toolset.

Findings

LLMs show varied capabilities in complex tool utilization

Independent natural language planning improves problem-solving accuracy

UltraTool provides new insights into LLMs' real-world tool use

Abstract

The recent trend of using Large Language Models (LLMs) as tool agents in real-world applications underscores the necessity for comprehensive evaluations of their capabilities, particularly in complex scenarios involving planning, creating, and using tools. However, existing benchmarks typically focus on simple synthesized queries that do not reflect real-world complexity, thereby offering limited perspectives in evaluating tool utilization. To address this issue, we present UltraTool, a novel benchmark designed to improve and evaluate LLMs' ability in tool utilization within real-world scenarios. UltraTool focuses on the entire process of using tools - from planning and creating to applying them in complex tasks. It emphasizes real-world complexities, demanding accurate, multi-step planning for effective problem-solving. A key feature of UltraTool is its independent evaluation of planning with natural language, which happens before tool usage and simplifies the task solving by mapping out the intermediate steps. Thus, unlike previous work, it eliminates the restriction of pre-defined toolset. Through extensive experiments on various LLMs, we offer novel insights into the evaluation of capabilities of LLMs in tool utilization, thereby contributing a fresh perspective to this rapidly evolving field. The benchmark is publicly available at https://github.com/JoeYing1019/UltraTool.