Brevity is the soul of wit: Pruning long files for code generation

TL;DR

This paper demonstrates that pruning long code files is an effective and simple data filtering heuristic for fine-tuning large language models for code generation, improving efficiency and performance.

Contribution

It introduces a heuristic method of pruning long files for data curation in code generation, outperforming embedding-based filtering in certain regimes.

Findings

Pruning long files improves training efficiency by up to 2x.

Pruning long files yields a 3.5% absolute performance boost on HumanEval.

Embedding-based methods are confounded by file length.

Abstract

Data curation is commonly considered a "secret-sauce" for LLM training, with higher quality data usually leading to better LLM performance. Given the scale of internet-scraped corpora, data pruning has become a larger and larger focus. Specifically, many have shown that de-duplicating data, or sub-selecting higher quality data, can lead to efficiency or performance improvements. Generally, three types of methods are used to filter internet-scale corpora: embedding-based, heuristic-based, and classifier-based. In this work, we contrast the former two in the domain of finetuning LLMs for code generation. We find that embedding-based methods are often confounded by length, and that a simple heuristic--pruning long files--outperforms other methods in compute-limited regimes. Our method can yield up to a 2x efficiency benefit in training (while matching performance) or a 3.5% absolute performance improvement on HumanEval (while matching compute). However, we find that perplexity on held-out long files can increase, begging the question of whether optimizing data mixtures for common coding benchmarks (HumanEval, MBPP) actually best serves downstream use cases. Overall, we hope our work builds useful intuitions about code data (specifically, the low quality of extremely long code files) provides a compelling heuristic-based method for data pruning, and brings to light questions in how we evaluate code generation models.