Interactive Evaluation of Large Language Models for Multi-Requirement Software Engineering Tasks

TL;DR

This paper introduces an interactive, feedback-driven evaluation framework for large language models tackling complex multi-requirement software engineering tasks, providing nuanced insights beyond static benchmarks.

Contribution

It presents a novel dynamic evaluation protocol using structured dialogues and hints, enhancing the assessment of LLMs' capabilities in complex programming tasks.

Findings

Interactive evaluation reveals model strengths and weaknesses.

Hints improve model performance and diagnostic insights.

Dynamic assessment surpasses static benchmarks in complexity understanding.

Abstract

Standard single-turn, static benchmarks fall short in evaluating the nuanced capabilities of Large Language Models (LLMs) on complex tasks such as software engineering. In this work, we propose a novel interactive evaluation framework that assesses LLMs on multi-requirement programming tasks through structured, feedback-driven dialogue. Each task is modeled as a requirement dependency graph, and an ``interviewer'' LLM, aware of the ground-truth solution, provides minimal, targeted hints to an ``interviewee'' model to help correct errors and fulfill target constraints. This dynamic protocol enables fine-grained diagnostic insights into model behavior, uncovering strengths and systematic weaknesses that static benchmarks fail to measure. We build on DevAI, a benchmark of 55 curated programming tasks, by adding ground-truth solutions and evaluating the relevance and utility of interviewer hints through expert annotation. Our results highlight the importance of dynamic evaluation in advancing the development of collaborative code-generating agents.