REST API Testing in DevOps: A Study on an Evolving Healthcare IoT Application

TL;DR

This study evaluates automated REST API testing tools in a healthcare IoT DevOps environment, revealing their effectiveness and limitations in regression testing for complex, evolving healthcare applications.

Contribution

The paper provides an empirical comparison of five REST API testing tools on a real-world healthcare IoT application within a DevOps context.

Findings

All tools detected failures and faults but with high overhead.

Coverage reached up to 84%, but many tests failed to detect failures.

Significant regressions and failures highlight challenges in automated testing.

Abstract

Healthcare Internet of Things (IoT) applications often integrate various third-party healthcare applications and medical devices through REST APIs, resulting in complex and interdependent networks of REST APIs. Oslo City's healthcare department collaborates with various industry partners to develop such healthcare IoT applications enriched with a diverse set of REST APIs. Following the DevOps process, these REST APIs continuously evolve to accommodate evolving needs such as new features, services, and devices. Oslo City's primary goal is to utilize automated solutions for continuous testing of these REST APIs at each evolution stage, thereby ensuring their dependability. Although the literature offers various automated REST API testing tools, their effectiveness in regression testing of the evolving REST APIs of healthcare IoT applications within a DevOps context remains undetermined. This paper evaluates state-of-the-art and well-established REST API testing tools, specifically, RESTest, EvoMaster, Schemathesis, RESTler, and RestTestGen, for the regression testing of a real-world healthcare IoT application, considering failures, faults, coverage, regressions, and cost. We conducted experiments using all accessible REST APIs (17 APIs with 120 endpoints), and 14 releases evolved during DevOps. Overall, all tools generated tests leading to several failures, 18 potential faults, up to 84% coverage, and 23 regressions. Over 70% of tests generated by all tools fail to detect failures, resulting in significant overhead.