LLMREI: Automating Requirements Elicitation Interviews with LLMs

TL;DR

This paper presents LLMREI, an AI-powered chatbot that automates requirements elicitation interviews, reducing human bias and resource needs, and demonstrating comparable error rates and effective context-aware questioning in simulated stakeholder interactions.

Contribution

Introduction of LLMREI, a novel chatbot system that automates requirements elicitation interviews using prompting techniques, with evaluation showing its effectiveness and potential for large-scale stakeholder engagement.

Findings

LLMREI makes similar errors to human interviewers.

It effectively extracts relevant requirements.

Demonstrates strong context-dependent questioning ability.

Abstract

Requirements elicitation interviews are crucial for gathering system requirements but heavily depend on skilled analysts, making them resource-intensive, susceptible to human biases, and prone to miscommunication. Recent advancements in Large Language Models present new opportunities for automating parts of this process. This study introduces LLMREI, a chat bot designed to conduct requirements elicitation interviews with minimal human intervention, aiming to reduce common interviewer errors and improve the scalability of requirements elicitation. We explored two main approaches, zero-shot prompting and least-to-most prompting, to optimize LLMREI for requirements elicitation and evaluated its performance in 33 simulated stakeholder interviews. A third approach, fine-tuning, was initially considered but abandoned due to poor performance in preliminary trials. Our study assesses the chat bot's effectiveness in three key areas: minimizing common interview errors, extracting relevant requirements, and adapting its questioning based on interview context and user responses. Our findings indicate that LLMREI makes a similar number of errors compared to human interviewers, is capable of extracting a large portion of requirements, and demonstrates a notable ability to generate highly context-dependent questions. We envision the greatest benefit of LLMREI in automating interviews with a large number of stakeholders.