Harvesting Production GraphQL Queries to Detect Schema Faults

TL;DR

This paper introduces AutoGraphQL, a method that leverages production GraphQL queries to automatically generate tests, effectively detecting schema faults and improving API reliability in real-world applications.

Contribution

The paper presents a novel approach using logged production queries to generate schema-validating tests, addressing the gap in testing techniques for GraphQL APIs.

Findings

AutoGraphQL covers 26.9% of Saleor schema, including untested parts.

It covers 48.7% of Frontapp schema and detects 8 schema faults.

The approach successfully generates meaningful tests from real user interactions.

Abstract

GraphQL is a new paradigm to design web APIs. Despite its growing popularity, there are few techniques to verify the implementation of a GraphQL API. We present a new testing approach based on GraphQL queries that are logged while users interact with an application in production. Our core motivation is that production queries capture real usages of the application, and are known to trigger behavior that may not be tested by developers. For each logged query, a test is generated to assert the validity of the GraphQL response with respect to the schema. We implement our approach in a tool called AutoGraphQL, and evaluate it on two real-world case studies that are diverse in their domain and technology stack: an open-source e-commerce application implemented in Python called Saleor, and an industrial case study which is a PHP-based finance website called Frontapp. AutoGraphQL successfully generates test cases for the two applications. The generated tests cover 26.9% of the Saleor schema, including parts of the API not exercised by the original test suite, as well as 48.7% of the Frontapp schema, detecting 8 schema faults, thanks to production queries.