Dr. Boot: Bootstrapping Program Synthesis Language Models to Perform Repairing

TL;DR

This paper introduces a bootstrapping algorithm for program synthesis models that emphasizes repairing capabilities, outperforming traditional fine-tuning and aligning more closely with human iterative development processes.

Contribution

The paper presents a novel bootstrapping method that enhances program synthesis models' repairing skills, matching larger models' performance with smaller ones and improving non-repairing accuracy.

Findings

Bootstrapping outperforms regular fine-tuning.

Bootstrapped models match larger fine-tuned models in performance.

Repairing improves non-repairing performance during inference.

Abstract

Language models for program synthesis are usually trained and evaluated on programming competition datasets (MBPP, APPS). However, these datasets are limited in size and quality, while these language models are extremely data hungry. Additionally, the language models have a misaligned program synthesis process compared to humans. While humans iteratively develop code with the help of a compiler, most program synthesis models currently produce code in one go. To solve these issues, we introduce a bootstrapping algorithm for program synthesis, that supports teaching models how to repair. We show that bootstrapping consistently outperforms regular fine-tuning. Compared to other work, our bootstrapped model performs on par with fine-tuned models that are 68\% larger. Notably, bootstrapping with repairing also improves non-repairing performance compared to regular bootstrapping during inference. However, on our models, repairing during inference is likely inferior to simply sampling the same number of solutions. Furthermore, we find that there are issues with the example test cases in the training portion of the APPS dataset that are valuable to the community, as many repairing and reinforcement learning methods rely on them.