Testing Medical Rules Web Services in Practice

TL;DR

This paper evaluates automated testing tools for the Norwegian Cancer Registry's medical rule engine, demonstrating their effectiveness in code and rule coverage, fault detection, and recommending EvoGURI for practical use.

Contribution

It introduces and empirically assesses EvoGURI, a novel CRN-specific testing tool, alongside existing tools, for improving testing of medical rule systems.

Findings

All tools achieved similar code coverage and error detection.

EvoGURI and EvoMaster's black-box mode covered most rules.

EvoGURI and white-box tools identified the most faults.

Abstract

The Cancer Registry of Norway (CRN) collects and processes cancer-related data for patients in Norway. For this, it employs a sociotechnical software system that evolves with changing requirements and medical standards. The current practice is to manually test CRN's system to prevent faults and ensure its dependability. This paper focuses on automatically testing GURI, the CRN's medical rule engine, using a system-level testing tool, EvoMaster, in both its black-box and white-box modes, and a novel CRN-specific EvoMaster-based tool, EvoGURI. We empirically evaluate the tools' effectiveness regarding code coverage, errors found, domain-specific rule coverage, and ability to identify artificial faults ten versions of GURI. Our results show that all the tools achieve similar code coverage and identified a similar number of errors. For rule coverage, EvoGURI and EvoMaster's black-box mode produce test suites that cover the highest number of rules with Pass, Fail, and Warning results. The test suites of EvoGURI and two EvoMaster white-box tools identify the most faults in a mutation testing experiment. Based on our findings, we recommend using EvoGURI in CRN's current practice. Finally, we present key takeaways and outline open research questions for the research community.