When Benchmarks Talk: Re-Evaluating Code LLMs with Interactive Feedback

TL;DR

This paper introduces an interactive evaluation method for code LLMs that assesses how models incorporate user feedback during collaboration, revealing performance variations and behavioral impacts not captured by static benchmarks.

Contribution

It presents a novel interactive evaluation pipeline that perturbs static benchmarks to simulate user interactions, providing insights into model behavior and performance in collaborative coding scenarios.

Findings

Interaction significantly alters model rankings across datasets.

Models are robust to feedback containing errors.

Feedback type influences model responses and edit priorities.

Abstract

Programming is a fundamentally interactive process, yet coding assistants are often evaluated using static benchmarks that fail to measure how well models collaborate with users. We introduce an interactive evaluation pipeline to examine how LLMs incorporate different types of feedback in a collaborative setting. Specifically, we perturb static coding benchmarks so that the code model must interact with a simulated user to retrieve key information about the problem. We find that interaction significantly affects model performance, as the relative rankings of 10 models across 3 datasets often vary between static and interactive settings, despite models being fairly robust to feedback that contains errors. We also observe that even when different feedback types are equally effective with respect to performance, they can impact model behaviors such as (1) how models respond to higher- vs. lower-quality feedback and (2) whether models prioritize aesthetic vs. functional edits. Our work aims to "re-evaluate" model coding capabilities through an interactive lens toward bridging the gap between existing evaluations and real-world usage.