ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programming

TL;DR

This paper introduces ELABORATION, a comprehensive benchmark for evaluating human-LLM collaboration in competitive programming, including a new taxonomy of human feedback, a specialized dataset, and an assessment framework to identify strengths and weaknesses.

Contribution

It presents the first taxonomy of human feedback in programming, a dedicated dataset for human-LLM collaboration, and a benchmark for thorough evaluation of methods.

Findings

Identified key strengths and weaknesses of current approaches.

Provided a new dataset annotated for human feedback simulation.

Established a benchmark for future research in human-LLM collaborative programming.

Abstract

While recent research increasingly emphasizes the value of human-LLM collaboration in competitive programming and proposes numerous empirical methods, a comprehensive understanding remains elusive due to the fragmented nature of existing studies and their use of diverse, application-specific human feedback. Thus, our work serves a three-fold purpose: First, we present the first taxonomy of human feedback consolidating the entire programming process, which promotes fine-grained evaluation. Second, we introduce ELABORATIONSET, a novel programming dataset specifically designed for human-LLM collaboration, meticulously annotated to enable large-scale simulated human feedback and facilitate costeffective real human interaction studies. Third, we introduce ELABORATION, a novel benchmark to facilitate a thorough assessment of human-LLM competitive programming. With ELABORATION, we pinpoint strengthes and weaknesses of existing methods, thereby setting the foundation for future improvement. Our code and dataset are available at https://github.com/SCUNLP/ELABORATION