SpecOps: A Fully Automated AI Agent Testing Framework in Real-World GUI Environments

TL;DR

SpecOps is a fully automated framework that evaluates real-world GUI-based AI agents using specialized LLMs, improving testing accuracy, bug detection, and efficiency over existing methods.

Contribution

It introduces a novel, structured architecture for automated testing of real-world GUI agents, addressing key challenges and outperforming baseline systems.

Findings

Identified 164 true bugs with high accuracy (F1=0.89)

Outperformed baselines in planning, execution, and bug detection

Cost under 0.73 USD with under eight minutes per test

Abstract

Autonomous AI agents powered by large language models (LLMs) are increasingly deployed in real-world applications, where reliable and robust behavior is critical. However, existing agent evaluation frameworks either rely heavily on manual efforts, operate within simulated environments, or lack focus on testing complex, multimodal, real-world agents. We introduce SpecOps, a novel, fully automated testing framework designed to evaluate GUI-based AI agents in real-world environments. SpecOps decomposes the testing process into four specialized phases - test case generation, environment setup, test execution, and validation - each handled by a distinct LLM-based specialist agent. This structured architecture addresses key challenges including end-to-end task coherence, robust error handling, and adaptability across diverse agent platforms including CLI tools, web apps, and browser extensions. In comprehensive evaluations across five diverse real-world agents, SpecOps outperforms baselines including general-purpose agentic systems such as AutoGPT and LLM-crafted automation scripts in planning accuracy, execution success, and bug detection effectiveness. SpecOps identifies 164 true bugs in the real-world agents with an F1 score of 0.89. With a cost of under 0.73 USD and a runtime of under eight minutes per test, it demonstrates its practical viability and superiority in automated, real-world agent testing.