ToolSpectrum : Towards Personalized Tool Utilization for Large Language Models

TL;DR

This paper introduces ToolSpectrum, a benchmark for evaluating personalized tool utilization in large language models, emphasizing the importance of context-aware personalization involving user profiles and environmental factors.

Contribution

The paper formalizes key dimensions of personalization in tool selection for LLMs and provides an extensive benchmark to evaluate their capabilities in this area.

Findings

Personalized tool utilization improves user experience.

State-of-the-art LLMs struggle to jointly reason about user and environmental factors.

Current models often prioritize one personalization dimension over the other.

Abstract

While integrating external tools into large language models (LLMs) enhances their ability to access real-time information and domain-specific services, existing approaches focus narrowly on functional tool selection following user instructions, overlooking the context-aware personalization in tool selection. This oversight leads to suboptimal user satisfaction and inefficient tool utilization, particularly when overlapping toolsets require nuanced selection based on contextual factors. To bridge this gap, we introduce ToolSpectrum, a benchmark designed to evaluate LLMs' capabilities in personalized tool utilization. Specifically, we formalize two key dimensions of personalization, user profile and environmental factors, and analyze their individual and synergistic impacts on tool utilization. Through extensive experiments on ToolSpectrum, we demonstrate that personalized tool utilization significantly improves user experience across diverse scenarios. However, even state-of-the-art LLMs exhibit the limited ability to reason jointly about user profiles and environmental factors, often prioritizing one dimension at the expense of the other. Our findings underscore the necessity of context-aware personalization in tool-augmented LLMs and reveal critical limitations for current models. Our data and code are available at https://github.com/Chengziha0/ToolSpectrum.