A Benchmark Dataset And LLMs Comparison For NFR Classification With Explainable AI

TL;DR

This paper introduces a new dataset for NFR classification, compares multiple LLMs on this task, and demonstrates that models like Gemma-2 and Phi-3 achieve high accuracy and explainability in classifying non-functional requirements.

Contribution

It provides a comprehensive NFR dataset and evaluates several LLMs, highlighting the effectiveness of Gemma-2 and Phi-3 for automated NFR classification with explainability.

Findings

Gemma-2 achieved the highest precision and recall.

Phi-3 had the best lime hit score.

The dataset enhancement improved model understanding of technical and user needs.

Abstract

Non-Functional Requirements (NFRs) play a critical role in determining the overall quality and user satisfaction of software systems. Accurately identifying and classifying NFRs is essential to ensure that software meets performance, usability, and reliability expectations. However, manual identification of NFRs from documentation is time-consuming and prone to errors, necessitating automated solutions. Before implementing any automated solution, a robust and comprehensive dataset is essential. To build such a dataset, we collected NFRs from various Project Charters and Open Source Software Documentation. This enhanced the technical depth and usability of an already existing NFR dataset. We categorized NFRs into sub-classes and identified needs using widely used Large Language Models to facilitate automation. After classifying the NFRs, we compared the classification results of the selected LLMs: RoBERTa, CodeBERT, Gemma-2, Phi-3, Mistral-8B, and Llama-3.1-8B using various evaluation metrics, including precision, recall, F1-score, and lime scores. Among these models, Gemma-2 achieved the best results with a precision of 0.87, recall of 0.89, and F1-score of 0.88, alongside a lime hit score of 78 out of 80. Phi-3 closely followed with a precision of 0.85, recall of 0.87, F1-score of 0.86, and the highest lime hit score of 79. By improving the contextual foundation, this integration enhanced the model's comprehension of technical aspects and user requirements.