AutoCoder: Enhancing Code Large Language Model with \textsc{AIEV-Instruct}

TL;DR

AutoCoder is a new large language model that outperforms GPT-4 Turbo on code generation benchmarks and features a versatile code interpreter, trained using a novel AIEV-Instruct method that leverages agent interaction and execution verification.

Contribution

AutoCoder introduces the first LLM surpassing GPT-4 Turbo in code generation performance and employs a new training approach, AIEV-Instruct, for creating execution-verified, versatile code datasets.

Findings

AutoCoder achieves 90.9% pass@1 on Human Eval, surpassing GPT-4 Turbo.

AutoCoder's code interpreter can install external packages, enhancing versatility.

AIEV-Instruct reduces reliance on proprietary models for dataset creation.

Abstract

We introduce AutoCoder, the first Large Language Model to surpass GPT-4 Turbo (April 2024) and GPT-4o in pass@1 on the Human Eval benchmark test ($\mathbf{90.9\%}$ vs. $\mathbf{90.2\%}$). In addition, AutoCoder offers a more versatile code interpreter compared to GPT-4 Turbo and GPT-4o. It's code interpreter can install external packages instead of limiting to built-in packages. AutoCoder's training data is a multi-turn dialogue dataset created by a system combining agent interaction and external code execution verification, a method we term \textbf{\textsc{AIEV-Instruct}} (Instruction Tuning with Agent-Interaction and Execution-Verified). Compared to previous large-scale code dataset generation methods, \textsc{AIEV-Instruct} reduces dependence on proprietary large models and provides execution-validated code dataset. The code and the demo video is available in \url{https://github.com/bin123apple/AutoCoder}.