LLMs in the Heart of Differential Testing: A Case Study on a Medical Rule Engine

TL;DR

This paper explores using large language models to generate tests for a medical rule engine in cancer data registration, revealing their effectiveness and limitations in identifying implementation issues.

Contribution

It introduces LLMeDiff, a novel LLM-based differential testing approach for medical rule engines, demonstrating its effectiveness in uncovering inconsistencies.

Findings

GPT-3.5 performs best in success and robustness

Hallucinates least among tested LLMs

Identified 22 rules with implementation inconsistencies

Abstract

The Cancer Registry of Norway (CRN) uses an automated cancer registration support system (CaReSS) to support core cancer registry activities, i.e, data capture, data curation, and producing data products and statistics for various stakeholders. GURI is a core component of CaReSS, which is responsible for validating incoming data with medical rules. Such medical rules are manually implemented by medical experts based on medical standards, regulations, and research. Since large language models (LLMs) have been trained on a large amount of public information, including these documents, they can be employed to generate tests for GURI. Thus, we propose an LLM-based test generation and differential testing approach (LLMeDiff) to test GURI. We experimented with four different LLMs, two medical rule engine implementations, and 58 real medical rules to investigate the hallucination, success, time efficiency, and robustness of the LLMs to generate tests, and these tests' ability to find potential issues in GURI. Our results showed that GPT-3.5 hallucinates the least, is the most successful, and is generally the most robust; however, it has the worst time efficiency. Our differential testing revealed 22 medical rules where implementation inconsistencies were discovered (e.g., regarding handling rule versions). Finally, we provide insights for practitioners and researchers based on the results.