Synthline: A Product Line Approach for Synthetic Requirements Engineering Data Generation using Large Language Models

TL;DR

Synthline introduces a product line approach using Large Language Models to generate synthetic requirements engineering data, improving ML model training and addressing data scarcity issues in RE.

Contribution

The paper presents a novel PL-based method for systematic synthetic RE data generation using LLMs, enhancing data diversity and utility for ML applications.

Findings

Synthetic data has lower diversity but is effective for training.

Hybrid datasets improve ML performance significantly.

Up to 85% precision improvement with combined data.

Abstract

While modern Requirements Engineering (RE) heavily relies on natural language processing and Machine Learning (ML) techniques, their effectiveness is limited by the scarcity of high-quality datasets. This paper introduces Synthline, a Product Line (PL) approach that leverages Large Language Models to systematically generate synthetic RE data for classification-based use cases. Through an empirical evaluation conducted in the context of using ML for the identification of requirements specification defects, we investigated both the diversity of the generated data and its utility for training downstream models. Our analysis reveals that while synthetic datasets exhibit less diversity than real data, they are good enough to serve as viable training resources. Moreover, our evaluation shows that combining synthetic and real data leads to substantial performance improvements. Specifically, hybrid approaches achieve up to 85% improvement in precision and a 2x increase in recall compared to models trained exclusively on real data. These findings demonstrate the potential of PL-based synthetic data generation to address data scarcity in RE. We make both our implementation and generated datasets publicly available to support reproducibility and advancement in the field.