Context-Adaptive Requirements Defect Prediction through Human-LLM Collaboration

TL;DR

This paper introduces a human-LLM collaboration approach for defect prediction in requirements engineering, enabling adaptive, context-aware classification that improves with stakeholder feedback and explanations.

Contribution

It proposes a novel adaptive defect prediction method using LLMs with feedback loops, surpassing traditional static models in contextual accuracy.

Findings

Rapid performance improvement with as few as 20 validated examples

Outperforms standard few-shot prompting and fine-tuned BERT models

Maintains high recall while adapting to stakeholder feedback

Abstract

Automated requirements assessment traditionally relies on universal patterns as proxies for defectiveness, implemented through rule-based heuristics or machine learning classifiers trained on large annotated datasets. However, what constitutes a "defect" is inherently context-dependent and varies across projects, domains, and stakeholder interpretations. In this paper, we propose a Human-LLM Collaboration (HLC) approach that treats defect prediction as an adaptive process rather than a static classification task. HLC leverages LLM Chain-of-Thought reasoning in a feedback loop: users validate predictions alongside their explanations, and these validated examples adaptively guide future predictions through few-shot learning. We evaluate this approach using the weak word smell on the QuRE benchmark of 1,266 annotated Mercedes-Benz requirements. Our results show that HLC effectively adapts to the provision of validated examples, with rapid performance gains from as few as 20 validated examples. Incorporating validated explanations, not just labels, enables HLC to substantially outperform both standard few-shot prompting and fine-tuned BERT models while maintaining high recall. These results highlight how the in-context and Chain-of-Thought learning capabilities of LLMs enable adaptive classification approaches that move beyond one-size-fits-all models, creating opportunities for tools that learn continuously from stakeholder feedback.