ReqElicitGym: An Evaluation Environment for Interview Competence in Conversational Requirements Elicitation

TL;DR

ReqElicitGym is a new automated environment for evaluating how well conversational AI models can perform requirements elicitation, revealing current limitations in LLMs' interview competence across diverse scenarios.

Contribution

It introduces ReqElicitGym, a comprehensive evaluation environment with a dataset, oracle user, and evaluator, enabling systematic and reproducible assessment of LLMs' interview skills in requirements elicitation.

Findings

Current LLMs elicit less than half of implicit requirements.

LLMs perform better with interaction and content requirements than style.

Elicitation questions tend to appear in later dialogue turns.

Abstract

With the rapid improvement of LLMs' coding capabilities, the bottleneck of LLM-based automated software development is shifting from generating correct code to eliciting users' requirements. Despite growing interest, the interview competence of LLMs in conversational requirements elicitation remains fully underexplored. Existing evaluations often depend on a few scenarios, real user interaction, and subjective human scoring, which hinders systematic and quantitative comparison. To address these challenges, we propose ReqElicitGym, an interactive and automatic evaluation environment for assessing interview competence in conversational requirements elicitation. Specifically, ReqElicitGym introduces a new evaluation dataset and designs both an interactive oracle user and a task evaluator. The dataset contains 101 website requirements elicitation scenarios spanning 10 application types. Both the oracle user and the task evaluator achieve high agreement with real users and expert judgment. Using our ReqElicitGym, any automated conversational requirements elicitation approach (e.g., LLM-based agents) can be evaluated in a reproducible and quantitative manner through interaction with the environment. Based on our ReqElicitGym, we conduct a systematic empirical study on seven representative LLMs, and the results show that current LLMs still exhibit limited interview competence in uncovering implicit requirements. Particularly, they elicit less than half of the users' implicit requirements, and their effective elicitation questions often emerge in later turns of the dialogue. Besides, we found LLMs can elicit interaction and content implicit requirements, but consistently struggle with style-related requirements. We believe ReqElicitGym will facilitate the evaluation and development of automated conversational requirements elicitation.