Multi-Agent LLM Committees for Autonomous Software Beta Testing

TL;DR

This paper introduces a multi-agent LLM committee framework for autonomous software beta testing, significantly improving success rates, UI understanding, and bug detection over single-agent approaches, with real-time performance and open-source tools.

Contribution

It presents a novel multi-agent committee approach combining diverse LLMs and visual understanding for effective, real-time, autonomous software testing, outperforming single-agent baselines.

Findings

Achieves 89.5% overall task success rate in testing scenarios.

Multi-agent configurations reach up to 100% success, outperforming 78% of single-agent baselines.

Attains 0.91 F1 score in bug detection, surpassing previous methods.

Abstract

Manual software beta testing is costly and time-consuming, while single-agent large language model (LLM) approaches suffer from hallucinations and inconsistent behavior. We propose a multi-agent committee framework in which diverse vision-enabled LLMs collaborate through a three-round voting protocol to reach consensus on testing actions. The framework combines model diversity, persona-driven behavioral variation, and visual user interface understanding to systematically explore web applications. Across 84 experimental runs with 9 testing personas and 4 scenarios, multi-agent committees achieve an 89.5 percent overall task success rate. Configurations with 2 to 4 agents reach 91.7 to 100 percent success, compared to 78.0 percent for single-agent baselines, yielding improvements of 13.7 to 22.0 percentage points. At the action level, the system attains a 93.1 percent success rate with a median per-action latency of 0.71 seconds, enabling real-time and continuous integration testing. Vision-enabled agents successfully identify user interface elements, with navigation and reporting achieving 100 percent success and form filling achieving 99.2 percent success. We evaluate the framework on WebShop and OWASP benchmarks, achieving 74.7 percent success on WebShop compared to a 50.1 percent published GPT-3 baseline, and 82.0 percent success on OWASP Juice Shop security testing with coverage of 8 of the 10 OWASP Top 10 vulnerability categories. Across 20 injected regressions, the committee achieves an F1 score of 0.91 for bug detection, compared to 0.78 for single-agent baselines. The open-source implementation enables reproducible research and practical deployment of LLM-based software testing in CI/CD pipelines.