Explicating Tacit Regulatory Knowledge from LLMs to Auto-Formalize Requirements for Compliance Test Case Generation

TL;DR

This paper introduces RAFT, a framework that leverages multiple LLMs and an adaptive strategy to automatically formalize regulatory requirements and generate compliance test cases, significantly improving accuracy and efficiency in regulated domains.

Contribution

RAFT is the first framework to explicate tacit regulatory knowledge from multiple LLMs for auto-formalization and test case generation, reducing manual effort and outperforming existing methods.

Findings

RAFT achieves expert-level performance in multiple domains.

RAFT outperforms state-of-the-art methods in accuracy and efficiency.

The adaptive strategy effectively integrates knowledge from multiple LLMs.

Abstract

Compliance testing in highly regulated domains is crucial but largely manual, requiring domain experts to translate complex regulations into executable test cases. While large language models (LLMs) show promise for automation, their susceptibility to hallucinations limits reliable application. Existing hybrid approaches mitigate this issue by constraining LLMs with formal models, but still rely on costly manual modeling. To solve this problem, this paper proposes RAFT, a framework for requirements auto-formalization and compliance test generation via explicating tacit regulatory knowledge from multiple LLMs. RAFT employs an Adaptive Purification-Aggregation strategy to explicate tacit regulatory knowledge from multiple LLMs and integrate it into three artifacts: a domain meta-model, a formal requirements representation, and testability constraints. These artifacts are then dynamically injected into prompts to guide high-precision requirement formalization and automated test generation. Experiments across financial, automotive, and power domains show that RAFT achieves expert-level performance, substantially outperforms state-of-the-art (SOTA) methods while reducing overall generation and review time.