AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition

TL;DR

This paper introduces AgentNoiseBench, a framework for systematically evaluating the robustness of tool-using LLM agents under noisy real-world conditions, revealing their sensitivity to environmental perturbations.

Contribution

We develop a noise-injection pipeline for agent benchmarks and provide extensive evaluations showing how noise impacts LLM agent performance.

Findings

Models exhibit significant performance drops under noise.

Performance varies with different noise types and levels.

Current agents are sensitive to real-world environmental noise.

Abstract

Recent advances in large language models have enabled LLM-based agents to achieve strong performance on a variety of benchmarks. However, their performance in real-world deployments often that observed on benchmark settings, especially in complex and imperfect environments. This discrepancy largely arises because prevailing training and evaluation paradigms are typically built on idealized assumptions, overlooking the inherent stochasticity and noise present in real-world interactions. To bridge this gap, we introduce AgentNoiseBench, a framework for systematically evaluating the robustness of agentic models under noisy environments. We first conduct an in-depth analysis of biases and uncertainties in real-world scenarios and categorize environmental noise into two primary types: user-noise and tool-noise. Building on this analysis, we develop an automated pipeline that injects controllable noise into existing agent-centric benchmarks while preserving task solvability. Leveraging this pipeline, we perform extensive evaluations across a wide range of models with diverse architectures and parameter scales. Our results reveal consistent performance variations under different noise conditions, highlighting the sensitivity of current agentic models to realistic environmental perturbations.