CodeAlignBench: Assessing Code Generation Models on Developer-Preferred Code Adjustments

TL;DR

This paper introduces CodeAlignBench, a comprehensive multi-language benchmark for evaluating code generation models on developer-aligned code adjustments, emphasizing instruction-following and refinement capabilities beyond functional correctness.

Contribution

We present a new extensible benchmark that assesses instruction-following and refinement in code generation across multiple languages, addressing limitations of existing evaluation methods.

Findings

Models show varied performance across languages and tasks.

Benchmark reveals strengths and limitations of current code models.

Automated evaluation pipeline enables comprehensive analysis.

Abstract

As large language models become increasingly capable of generating code, evaluating their performance remains a complex and evolving challenge. Existing benchmarks primarily focus on functional correctness, overlooking the diversity of real-world coding tasks and developer expectations. To this end, we introduce a multi-language benchmark that evaluates LLM instruction-following capabilities and is extensible to operate on any set of standalone coding problems. Our benchmark evaluates instruction following in two key settings: adherence to pre-defined constraints specified with the initial problem, and the ability to perform refinements based on follow-up instructions. For this paper's analysis, we empirically evaluated our benchmarking pipeline with programming tasks from LiveBench, that are also automatically translated from Python into Java and JavaScript. Our automated benchmark reveals that models exhibit differing levels of performance across multiple dimensions of instruction-following. Our benchmarking pipeline provides a more comprehensive evaluation of code generation models, highlighting their strengths and limitations across languages and generation goals.