Grey-Box Fuzzing in Constrained Ultra-Large Systems: Lessons for SE Community

TL;DR

This paper introduces SandBoxFuzz, a scalable grey-box fuzzing tool designed for ultra-large microservices systems, improving coverage and efficiency in constrained industrial environments.

Contribution

We present SandBoxFuzz, a novel fuzzing approach that uses aspect-oriented programming and runtime reflection to enable effective testing of large-scale microservices.

Findings

7.5% increase in branch coverage

Identified 1,850 additional exceptions

Reduced setup time from hours to minutes

Abstract

Testing ultra-large microservices-based FinTech systems presents significant challenges, including restricted access to production environments, complex dependencies, and stringent security constraints. We propose SandBoxFuzz, a scalable grey-box fuzzing technique that addresses these limitations by leveraging aspect-oriented programming and runtime reflection to enable dynamic specification mining, generating targeted inputs for constrained environments. SandBoxFuzz also introduces a log-based coverage mechanism, seamlessly integrated into the build pipeline, eliminating the need for runtime coverage agents that are often infeasible in industrial settings. SandBoxFuzz has been successfully deployed to Ant Group's production line and, compared to an initial solution built on a state-of-the-art fuzzing framework, it demonstrates superior performance in their microservices software. SandBoxFuzz achieves a 7.5% increase in branch coverage, identifies 1,850 additional exceptions, and reduces setup time from hours to minutes, highlighting its effectiveness and practical utility in a real-world industrial environment. By open-sourcing SandBoxFuzz, we provide a practical and effective tool for researchers and practitioners to test large-scale microservices systems.