CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generation

TL;DR

CoCoEvo introduces a co-evolutionary framework that simultaneously evolves programs and test cases using LLMs, eliminating the need for pre-defined test cases and improving code generation quality.

Contribution

This paper presents a novel co-evolution approach that generates both programs and test cases from natural language, enhancing automated code generation without relying on pre-existing test data.

Findings

Outperforms existing code generation methods on multiple LLMs

Achieves state-of-the-art results in automated testing and code quality

Demonstrates the effectiveness of co-evolutionary strategies in programming tasks

Abstract

Large Language Models (LLMs) have shown remarkable performance in automated code generation. However, existing approaches often rely heavily on pre-defined test cases, which become impractical in scenarios where such cases are unavailable. While prior works explore filtering techniques between programs and test cases, they overlook the refinement of test cases. To address this limitation, we introduce CoCoEvo, a novel LLM-based co-evolution framework that simultaneously evolves programs and test cases. CoCoEvo eliminates the dependency on pre-defined test cases by generating both programs and test cases directly from natural language problem descriptions and function headers. The framework employs specialized evolutionary operators, including LLM-based crossover and mutation operators for program evolution, along with an additional test case generation operator for test case evolution. Additionally, we propose optimization strategies such as a crossover rate scheduler to balance exploration and convergence, and a multi-objective optimization method for test case selection. Experimental results on multiple state-of-the-art LLMs demonstrate that CoCoEvo surpasses existing methods, achieving state-of-the-art performance in automated code generation and testing. These results underscore the potential of co-evolutionary techniques in advancing the field of automated programming.