CodeShell Technical Report

TL;DR

This paper introduces CodeShell-Base, a 7-billion-parameter code-focused language model with advanced architecture and high-quality data, demonstrating superior performance in code comprehension and generation tasks across multiple programming languages.

Contribution

The paper presents a novel 7-billion-parameter model with integrated architectural enhancements and a comprehensive data pre-processing pipeline, achieving state-of-the-art results in code understanding and generation.

Findings

Outperforms CodeLlama on Humaneval after 500 billion tokens training

Curated 100 billion high-quality GitHub code data

Demonstrates strong performance across Python, Java, C++ datasets

Abstract

Code large language models mark a pivotal breakthrough in artificial intelligence. They are specifically crafted to understand and generate programming languages, significantly boosting the efficiency of coding development workflows. In this technical report, we present CodeShell-Base, a seven billion-parameter foundation model with 8K context length, showcasing exceptional proficiency in code comprehension. By incorporating Grouped-Query Attention and Rotary Positional Embedding into GPT-2, CodeShell-Base integrates the structural merits of StarCoder and CodeLlama and forms its unique architectural design. We then carefully built a comprehensive data pre-processing process, including similar data deduplication, perplexity-based data filtering, and model-based data filtering. Through this process, We have curated 100 billion high-quality pre-training data from GitHub. Benefiting from the high-quality data, CodeShell-Base outperforms CodeLlama in Humaneval after training on just 500 billion tokens (5 epochs). We have conducted extensive experiments across multiple language datasets, including Python, Java, and C++, and the results indicate that our model possesses robust foundational capabilities in code comprehension and generation.