CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing

TL;DR

This paper introduces a new dataset and empirical analysis of large language models for API function calling and routing, addressing multi-step, complex API interactions in chatbot systems.

Contribution

It presents a novel benchmarking dataset, evaluates state-of-the-art models, and proposes a hybrid API routing approach combining LLMs and fine-tuning.

Findings

Improved API execution accuracy in chatbot systems

Effective hybrid approach for API routing and parameter generation

Insights into model performance across task complexities

Abstract

API-driven chatbot systems are increasingly integral to software engineering applications, yet their effectiveness hinges on accurately generating and executing API calls. This is particularly challenging in scenarios requiring multi-step interactions with complex parameterization and nested API dependencies. Addressing these challenges, this work contributes to the evaluation and assessment of AI-based software development through three key advancements: (1) the introduction of a novel dataset specifically designed for benchmarking API function selection, parameter generation, and nested API execution; (2) an empirical evaluation of state-of-the-art language models, analyzing their performance across varying task complexities in API function generation and parameter accuracy; and (3) a hybrid approach to API routing, combining general-purpose large language models for API selection with fine-tuned models and prompt engineering for parameter generation. These innovations significantly improve API execution in chatbot systems, offering practical methodologies for enhancing software design, testing, and operational workflows in real-world software engineering contexts.