Automatic Property-based Testing of GraphQL APIs

TL;DR

This paper introduces an automated, property-based method for generating GraphQL queries to test APIs, effectively finding bugs and covering schemas quickly, addressing a gap in automated testing tools for GraphQL.

Contribution

It presents a novel property-based query generator for GraphQL APIs, enabling automated testing and bug detection based on schema analysis.

Findings

Effectively finds real bugs in GraphQL APIs

Covers complete schemas in seconds

Seeds 73% of known faults successfully

Abstract

In recent years, GraphQL has become a popular way to expose web APIs. With its raise of adoption in industry, the quality of GraphQL APIs must be also assessed, as with any part of a software system, and preferably in an automated manner. However, there is currently a lack of methods to automatically generate tests to exercise GraphQL APIs. In this paper, we propose a method for automatically producing GraphQL queries to test GraphQL APIs. This is achieved using a property-based approach to create a generator for queries based on the GraphQL schema of the system under test. Our evaluation on a real world software system shows that this approach is both effective, in terms of finding real bugs, and efficient, as a complete schema can be covered in seconds. In addition, we evaluate the fault finding capability of the method when seeding known faults. 73% of the seeded faults where found, with room for improvements with regards to domain specific behavior, a common oracle challenge in automatic test generation.