ScaleCall -- Agentic Tool Calling at Scale for Fintech: Challenges, Methods, and Deployment Insights

TL;DR

This paper presents ScaleCall, a framework for tool calling in fintech enterprise environments, analyzing retrieval methods and deployment challenges to optimize accuracy and efficiency within regulatory constraints.

Contribution

It introduces ScaleCall, a novel tool-calling framework tailored for regulated enterprise settings, and provides a comprehensive evaluation of retrieval methods and deployment insights.

Findings

Embedding-based retrieval offers low latency for large tool sets.

Listwise ranking improves disambiguation of overlapping tools.

Hybrid approaches balance accuracy and efficiency in deployment.

Abstract

While Large Language Models (LLMs) excel at tool calling, deploying these capabilities in regulated enterprise environments such as fintech presents unique challenges due to on-premises constraints, regulatory compliance requirements, and the need to disambiguate large, functionally overlapping toolsets. In this paper, we present a comprehensive study of tool retrieval methods for enterprise environments through the development and deployment of ScaleCall, a prototype tool-calling framework within Mastercard designed for orchestrating internal APIs and automating data engineering workflows. We systematically evaluate embedding-based retrieval, prompt-based listwise ranking, and hybrid approaches, revealing that method effectiveness depends heavily on domain-specific factors rather than inherent algorithmic superiority. Through empirical investigation on enterprise-derived benchmarks, we find that embedding-based methods offer superior latency for large tool repositories, while listwise ranking provides better disambiguation for overlapping functionalities, with hybrid approaches showing promise in specific contexts. We integrate our findings into ScaleCall's flexible architecture and validate the framework through real-world deployment in Mastercard's regulated environment. Our work provides practical insights into the trade-offs between retrieval accuracy, computational efficiency, and operational requirements, contributing to the understanding of tool-calling system design for enterprise applications in regulated industries.