White-Box and Black-Box Fuzzing for GraphQL APIs

TL;DR

This paper presents an automated framework for testing GraphQL APIs using white-box and black-box methods, employing evolutionary and random search techniques, respectively, to improve fault detection and code coverage.

Contribution

It introduces a comprehensive testing framework for GraphQL APIs that integrates white-box and black-box testing with evolutionary and random search strategies.

Findings

White-box testing with evolutionary search improves code coverage.

Black-box testing detects real faults in online APIs.

Framework is integrated into the open-source EvoMaster tool.

Abstract

The Graph Query Language (GraphQL) is a powerful language for APIs manipulation in web services. It has been recently introduced as an alternative solution for addressing the limitations of RESTful APIs. This paper introduces an automated solution for GraphQL APIs testing. We present a full framework for automated APIs testing, from the schema extraction to test case generation. In addition, we consider two kinds of testing: white-box and black-box testing. The white-box testing is performed when the source code of the GraphQL API is available. Our approach is based on evolutionary search. Test cases are evolved to intelligently explore the solution space while maximizing code coverage and fault-finding criteria. The black-box testing does not require access to the source code of the GraphQL API. It is therefore of more general applicability, albeit it has worse performance. In this context, we use a random search to generate GraphQL data. The proposed framework is implemented and integrated into the open-source EvoMaster tool. With enabled white-box heuristics, i.e., white-box mode, experiments on 7 open-source GraphQL APIs show statistically significant improvement of the evolutionary approach compared to the baseline random search. In addition, experiments on 31 online GraphQL APIs reveal the ability of the black-box mode to detect real faults.