OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement

TL;DR

OpenCodeInterpreter is an open-source system that combines code generation, execution, and iterative refinement, achieving performance close to GPT-4 and bridging the gap between open models and proprietary systems.

Contribution

It introduces a family of open-source code systems with execution and feedback integration, supported by a large multi-turn interaction dataset, enhancing code accuracy and refinement.

Findings

OpenCodeInterpreter-33B achieves 83.2% accuracy on HumanEval and MBPP.

Performance is close to GPT-4's 84.2%, especially with GPT-4 synthesized feedback.

The system significantly narrows the gap between open-source and proprietary code generation models.

Abstract

The introduction of large language models has significantly advanced code generation. However, open-source models often lack the execution capabilities and iterative refinement of advanced systems like the GPT-4 Code Interpreter. To address this, we introduce OpenCodeInterpreter, a family of open-source code systems designed for generating, executing, and iteratively refining code. Supported by Code-Feedback, a dataset featuring 68K multi-turn interactions, OpenCodeInterpreter integrates execution and human feedback for dynamic code refinement. Our comprehensive evaluation of OpenCodeInterpreter across key benchmarks such as HumanEval, MBPP, and their enhanced versions from EvalPlus reveals its exceptional performance. Notably, OpenCodeInterpreter-33B achieves an accuracy of 83.2 (76.4) on the average (and plus versions) of HumanEval and MBPP, closely rivaling GPT-4's 84.2 (76.2) and further elevates to 91.6 (84.6) with synthesized human feedback from GPT-4. OpenCodeInterpreter brings the gap between open-source code generation models and proprietary systems like GPT-4 Code Interpreter.