AutoTest: Evolutionary Code Solution Selection with Test Cases

TL;DR

AutoTest introduces an evolutionary approach that combines automated test case generation and code execution to improve code solution selection, achieving about 10% better accuracy on the HumanEval benchmark.

Contribution

It presents a novel method integrating large language models, automated testing, and genetic algorithms for optimized code solution selection.

Findings

10% improvement in pass@1 score on HumanEval

Effective use of genetic algorithms for code ranking

Enhanced solution accuracy over baseline methods

Abstract

With the development of code generation techniques, selecting the correct code solution from multiple candidate solutions has become a crucial task. This study proposes AutoTest, a novel technique that combines automated test case generation with code solution execution to optimize the selection process using an evolutionary genetic algorithm. Firstly, AutoTest utilizes large pre-trained language models such as codegen-16B, code-davinci-002, and incoder-6B to provide code solutions and their corresponding test cases. Then, by executing the code solutions and evaluating their performance on the test cases, a consensus set is formed. Fine-grained ranking is achieved through the selection, mutation, and crossover mechanisms based on the evolutionary genetic algorithm, with the adjustment of alpha and beta parameters. Finally, the best code solution is chosen. AutoTest demonstrates significant performance improvements on the HumanEval benchmark test. The HumanEval dataset consists of 164 programming problems, and AutoTest achieves approximately a 10% improvement over the baseline method in terms of pass@1 score.