Lessons from the Use of Natural Language Inference (NLI) in Requirements Engineering Tasks

TL;DR

This paper evaluates the effectiveness of Natural Language Inference (NLI) in automating requirements engineering tasks, demonstrating its superiority over traditional NLP and LLM-based approaches across various learning settings.

Contribution

The study provides a comprehensive experimental comparison showing NLI's advantages in requirements analysis and offers lessons on optimal learning settings for its application.

Findings

NLI outperforms classical NLP methods in requirements tasks.

NLI surpasses LLM-based and chatbot models in accuracy.

Certain learning settings enhance NLI's effectiveness.

Abstract

We investigate the use of Natural Language Inference (NLI) in automating requirements engineering tasks. In particular, we focus on three tasks: requirements classification, identification of requirements specification defects, and detection of conflicts in stakeholders' requirements. While previous research has demonstrated significant benefit in using NLI as a universal method for a broad spectrum of natural language processing tasks, these advantages have not been investigated within the context of software requirements engineering. Therefore, we design experiments to evaluate the use of NLI in requirements analysis. We compare the performance of NLI with a spectrum of approaches, including prompt-based models, conventional transfer learning, Large Language Models (LLMs)-powered chatbot models, and probabilistic models. Through experiments conducted under various learning settings including conventional learning and zero-shot, we demonstrate conclusively that our NLI method surpasses classical NLP methods as well as other LLMs-based and chatbot models in the analysis of requirements specifications. Additionally, we share lessons learned characterizing the learning settings that make NLI a suitable approach for automating requirements engineering tasks.