Investigating the Transferability of Code Repair for Low-Resource Programming Languages

TL;DR

This paper explores how techniques like distillation for code repair transfer between high-resource and low-resource programming languages, revealing that benefits are language-dependent and reasoning ability correlates weakly with repair success.

Contribution

It investigates the transferability of code repair distillation techniques across different resource settings and analyzes the relationship between reasoning and repair abilities.

Findings

Distilling code repair benefits vary by language resource level.

Weak correlation between reasoning ability and code correction success.

Low-resource languages show less benefit from code repair distillation.

Abstract

Large language models (LLMs) have shown remarkable performance on code generation tasks. A recent use case is iterative code repair, where an LLM fixes an incorrect program by rationalizing about errors and generating new code. Recent works augment the code repair process by integrating modern techniques such as chain-of-thought reasoning or distillation, but only study their benefits on high-resource languages like Python, and ignore low-resource languages like Perl. To address this gap of knowledge, we investigate the benefits of distilling code repair for both high and low resource languages to determine if the techniques that are effective in a high resource setting are also applicable in a low resource setting. Our evaluation shows that distilling the ability to repair code has language dependent benefits. To explain this behavior, we perform a further analysis and find that contrary to preexisting beliefs, the correlation between reasoning ability and code correction ability is weak. We hypothesize this weak correlation is magnified in low-resource settings where base models lack deep knowledge of a programming language, leading to wavering benefits of code repair.