DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMs

TL;DR

This paper introduces DSTC, a novel framework that uses only self-generated tests and code snippets to improve code language models through direct preference learning, eliminating the need for external annotations.

Contribution

DSTC is a new method that constructs reliable preference pairs from self-generated data, enhancing code model accuracy without external annotations or reward models.

Findings

Improves pass@1 score across multiple benchmarks

Reduces reliance on costly annotated datasets

Enhances model performance without external annotations

Abstract

Direct preference learning offers a promising and computation-efficient beyond supervised fine-tuning (SFT) for improving code generation in coding large language models (LMs). However, the scarcity of reliable preference data is a bottleneck for the performance of direct preference learning to improve the coding accuracy of code LMs. In this paper, we introduce \underline{\textbf{D}}irect Preference Learning with Only \underline{\textbf{S}}elf-Generated \underline{\textbf{T}}ests and \underline{\textbf{C}}ode (DSTC), a framework that leverages only self-generated code snippets and tests to construct reliable preference pairs such that direct preference learning can improve LM coding accuracy without external annotations. DSTC combines a minimax selection process and test-code concatenation to improve preference pair quality, reducing the influence of incorrect self-generated tests and enhancing model performance without the need for costly reward models. When applied with direct preference learning methods such as Direct Preference Optimization (DPO) and Kahneman-Tversky Optimization (KTO), DSTC yields stable improvements in coding accuracy (pass@1 score) across diverse coding benchmarks, including HumanEval, MBPP, and BigCodeBench, demonstrating both its effectiveness and scalability for models of various sizes. This approach autonomously enhances code generation accuracy across LLMs of varying sizes, reducing reliance on expensive annotated coding datasets.