Qwen2.5-Coder Technical Report

TL;DR

The Qwen2.5-Coder series introduces six advanced code-specific models trained on extensive data, achieving state-of-the-art performance across multiple benchmarks and supporting broader research and application in code intelligence.

Contribution

This report presents a new series of six models with improved code generation capabilities, trained on a large corpus, and evaluated on numerous benchmarks, outperforming previous models of similar size.

Findings

Achieved state-of-the-art results on over 10 code-related benchmarks.

Demonstrated superior performance compared to larger models of the same size.

Maintained general and math skills while excelling in code tasks.

Abstract

In this report, we introduce the Qwen2.5-Coder series, a significant upgrade from its predecessor, CodeQwen1.5. This series includes six models: Qwen2.5-Coder-(0.5B/1.5B/3B/7B/14B/32B). As a code-specific model, Qwen2.5-Coder is built upon the Qwen2.5 architecture and continues pretrained on a vast corpus of over 5.5 trillion tokens. Through meticulous data cleaning, scalable synthetic data generation, and balanced data mixing, Qwen2.5-Coder demonstrates impressive code generation capabilities while retaining general and math skills. These models have been evaluated on a wide range of code-related tasks, achieving state-of-the-art (SOTA) performance across more than 10 benchmarks, including code generation, completion, reasoning, and repair, consistently outperforming larger models of the same model size. We believe that the release of the Qwen2.5-Coder series will advance research in code intelligence and, with its permissive licensing, support wider adoption by developers in real-world applications.