WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuning

TL;DR

WaveCoder introduces a versatile instruction tuning approach with a new dataset, significantly enhancing large language models' ability to generalize across diverse complex code-related tasks.

Contribution

The paper presents WaveCoder, a series of Code LLMs trained with a novel, high-quality, multi-task instruction dataset, improving generalization in complex code tasks.

Findings

WaveCoder models outperform existing open-source models in task generalization.

WaveCoder-Ultra-6.7B achieves state-of-the-art results across multiple code tasks.

The approach enhances multi-task performance in code-related applications.

Abstract

Recent work demonstrates that, after instruction tuning, Code Large Language Models (Code LLMs) can obtain impressive capabilities to address a wide range of code-related tasks. However, current instruction tuning methods for Code LLMs mainly focus on the traditional code generation task, resulting in poor performance in complex multi-task scenarios. In this paper, we concentrate on multiple code-related tasks and present WaveCoder, a series of Code LLMs trained with Widespread And Versatile Enhanced instruction data. To enable the models to tackle complex code-related tasks, we propose a method to stably generate diverse, high-quality instruction data from open source code dataset in multi-task scenarios and obtain CodeSeaXDataset, a dataset comprising 19,915 instruction instances across 4 code-related tasks, which is aimed at improving the generalization ability of Code LLM. Our experiments demonstrate that WaveCoder models significantly outperform other open-source models in terms of the generalization ability across different code-related tasks. Moreover, WaveCoder-Ultra-6.7B presents the state-of-the-art generalization abilities on a wide range of code-related tasks.