Requirements Satisfiability with In-Context Learning

TL;DR

This paper explores using in-context learning with language models like GPT-4 to verify requirement satisfaction in system specifications, achieving high accuracy in a GDPR compliance scenario.

Contribution

It introduces a novel application of ICL for requirements verification, employing multiple prompt design patterns and evaluating their effectiveness on real-world privacy compliance tasks.

Findings

GPT-4 achieves 96.7% accuracy in requirement satisfaction verification.

Chain-of-thought prompting improves GPT-3.5 accuracy by 9%.

Inverting requirements enhances dissatisfaction detection to 97.2%.

Abstract

Language models that can learn a task at inference time, called in-context learning (ICL), show increasing promise in natural language inference tasks. In ICL, a model user constructs a prompt to describe a task with a natural language instruction and zero or more examples, called demonstrations. The prompt is then input to the language model to generate a completion. In this paper, we apply ICL to the design and evaluation of satisfaction arguments, which describe how a requirement is satisfied by a system specification and associated domain knowledge. The approach builds on three prompt design patterns, including augmented generation, prompt tuning, and chain-of-thought prompting, and is evaluated on a privacy problem to check whether a mobile app scenario and associated design description satisfies eight consent requirements from the EU General Data Protection Regulation (GDPR). The overall results show that GPT-4 can be used to verify requirements satisfaction with 96.7% accuracy and dissatisfaction with 93.2% accuracy. Inverting the requirement improves verification of dissatisfaction to 97.2%. Chain-of-thought prompting improves overall GPT-3.5 performance by 9.0% accuracy. We discuss the trade-offs among templates, models and prompt strategies and provide a detailed analysis of the generated specifications to inform how the approach can be applied in practice.