CodeEvo: Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback

TL;DR

CodeEvo presents an interaction-driven, iterative framework using two LLM agents to synthesize high-quality, instruction-code pairs, significantly improving code generation performance over existing methods.

Contribution

It introduces a novel hybrid, iterative feedback mechanism involving two LLM agents for more rigorous and effective code-centric data synthesis.

Findings

Models trained on CodeEvo data outperform baselines on code benchmarks.

The hybrid feedback mechanism improves data quality and synthesis efficiency.

Extensive analysis provides insights into effective code data generation.

Abstract

Acquiring high-quality instruction-code pairs is essential for training Large Language Models (LLMs) for code generation. Manually curated data is expensive and inherently limited in scale, motivating the development of code-centric synthesis methods. Yet, current approaches either focus on augmenting existing code or rely on predefined heuristics, both lacking rigorous data validation, which results in synthetic data that is ungrounded, repetitive, or overly simplistic. Inspired by collaborative programming practices, we propose CodeEvo, a framework that synthesizes code data through iterative interactions between two LLM agents: a Coder, which generates candidate code and test cases based on given instructions, and a Reviewer, which guides the synthesis process by producing new instructions and feedback. We further introduce a hybrid feedback mechanism that combines compiler determinism with the generative flexibility of agents, enabling automatic quality control throughout synthesis. Extensive experiments demonstrate that models fine-tuned on CodeEvo data significantly outperform established baselines across code generation benchmarks with various difficulties. In-depth analyses further provide insights from multiple perspectives into effective code-centric data synthesis.