CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planning

TL;DR

This paper introduces CATP-LLM, a framework that enables large language models to generate cost-aware tool plans, optimizing task performance while minimizing execution costs through a new planning language, reinforcement learning, and a novel dataset.

Contribution

The paper presents the first cost-aware tool planning framework for LLMs, including a planning language, a reinforcement learning fine-tuning method, and a new dataset for evaluation.

Findings

CATP-LLM outperforms GPT-4 in plan quality by up to 93.9%.

The framework effectively balances task performance and execution costs.

OpenCATP dataset enables benchmarking of cost-aware planning methods.

Abstract

Utilizing large language models (LLMs) for tool planning has emerged as a promising avenue for developing general AI systems, where LLMs automatically schedule external tools (e.g., vision models) to tackle complex tasks based on task descriptions. To push this paradigm toward practical applications, it is crucial for LLMs to consider tool execution costs (e.g., execution time) for tool planning. Unfortunately, prior studies overlook the tool execution costs, leading to the generation of expensive plans whose costs outweigh their benefits in terms of task performance. To fill this gap, we propose the Cost-Aware Tool Planning with LLMs (CATP-LLM) framework, which for the first time provides a coherent design to empower LLMs for cost-aware tool planning. Specifically, To facilitate efficient concurrent tool execution and cost reduction, we design a tool planning language to enhance the LLM for creating multi-branch non-sequential plans. Moreover, we propose a cost-aware offline reinforcement learning algorithm to fine-tune the LLM to optimize the performance-cost trade-off in tool planning. In the lack of public cost-related datasets, we further present OpenCATP, the first dataset for cost-aware planning, which comprises 11,100 evaluation samples from diverse tasks. Extensive experiments show that CATP-LLM outperforms GPT-4 even when using Llama2-7B as its backbone, with the average improvement of 1.5%-93.9% in terms of plan quality. Codes and dataset are available at: https://github.com/duowuyms/OpenCATP-LLM.