GenX: Mastering Code and Test Generation with Execution Feedback

TL;DR

GenX introduces a joint training approach for code and test generation models that leverages execution feedback, improving code correctness and test case quality without relying solely on pre-existing test data.

Contribution

A novel concurrent training method for code and test generation models that uses execution feedback and data augmentation strategies.

Findings

Outperforms baseline models on the APPS dataset

Effectively generates and augments test cases

Filters and synthesizes correct code solutions

Abstract

Recent advancements in language modeling have enabled the translation of natural language into code, and the use of execution feedback to improve code generation. However, these methods often rely heavily on pre-existing test cases, which may not always be available or comprehensive. In this work, we propose a novel approach that concurrently trains a code generation model and a test generation model, utilizing execution feedback to refine and enhance the performance of both. We introduce two strategies for test and code data augmentation and a new scoring function for code and test ranking. We experiment on the APPS dataset and demonstrate that our approach can effectively generate and augment test cases, filter and synthesize correct code solutions, and rank the quality of generated code and tests. The results demonstrate that our models, when iteratively trained with an increasing number of test cases and code solutions, outperform those trained on the original dataset.