Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models

TL;DR

This paper evaluates the robustness of multilingual tool calling in large language models, revealing language mismatch issues and limited effectiveness of current mitigation strategies across diverse languages.

Contribution

Introduces MLCL, a diagnostic benchmark for multilingual tool calling, and systematically analyzes failure modes and potential system strategies.

Findings

Language mismatch is a major failure mode in multilingual tool calling.

Current mitigation strategies reduce errors but do not match English performance.

Multilingual robustness remains a significant challenge for LLMs in tool invocation.

Abstract

Large Language Models (LLMs) are increasingly deployed as agents that invoke external tools through structured function calls. While recent work reports strong tool-calling performance under standard English-centric evaluations, the robustness of tool calling under multilingual user interactions remains underexplored. In this work, we introduce MLCL, a diagnostic benchmark, and conduct a systematic evaluation of multilingual tool calling across Chinese, Hindi, and the low-resource language Igbo. Through fine-grained error analysis, we show that many failures occur despite correct intent understanding and tool selection. We identify parameter value language mismatch as a dominant failure mode, where models generate semantically appropriate parameter values in the user's language, violating language-invariant execution conventions. We further evaluate several inference-time system strategies and find that while these strategies substantially reduce language-induced execution errors, none of them can fully recover English-level performance.