Client--Library Compatibility Testing with API Interaction Snapshots

TL;DR

This paper introduces Gilesi, a Java framework that records API interaction snapshots during client tests to detect behavioral breaking changes in libraries that traditional tests might miss.

Contribution

It proposes a novel snapshot-based approach for client--library compatibility testing that leverages existing client tests without requiring additional assertions.

Findings

Gilesi reliably detects BBCs missed by traditional client tests.

Snapshot comparison effectively identifies behavioral changes in library APIs.

Preliminary case study validates the approach's effectiveness.

Abstract

Modern software development heavily relies on third-party libraries to speed up development and enhance quality. As libraries evolve, they may break the tacit contract established with their clients by introducing behavioral breaking changes (BBCs) that alter run-time behavior and silently break client applications without being detected at compile time. Traditional regression tests on the client side often fail to detect such BBCs, either due to limited library coverage or weak assertions that do not sufficiently exercise the library's expected behavior. To address this issue, we propose a novel approach to client--library compatibility testing that leverages existing client tests in a novel way. Instead of relying on developer-written assertions, we propose recording the actual interactions at the API boundary during the execution of client tests (protocol, input and output values, exceptions, etc.). These sequences of API interactions are stored as snapshots which capture the exact contract expected by a client at a specific point in time. As the library evolves, we compare the original and new snapshots to identify perturbations in the contract, flag potential BBCs, and notify clients. We implement this technique in our prototype tool Gilesi, a Java framework that automatically instruments library APIs, records snapshots, and compares them. Through a preliminary case study on several client--library pairs with artificially seeded BBCs, we show that Gilesi reliably detects BBCs missed by client test suites.