From Reasoning to Code: GRPO Optimization for Underrepresented Languages

TL;DR

This paper presents a novel reinforcement learning approach using Group Relative Policy Optimization to improve code generation in underrepresented languages by explicitly incorporating reasoning steps, demonstrated with Prolog.

Contribution

The paper introduces GRPO-based training for small-scale LLMs to enhance reasoning and code accuracy in low-resource languages, a novel application of reinforcement learning in this context.

Findings

Significant improvements in reasoning quality and code correctness.

Effective integration of reasoning feedback into the training loop.

Demonstrated success with Prolog as a case study.

Abstract

Generating accurate and executable code using large language models (LLMs) is challenging for languages with limited public training data compared to popular languages such as Python. This paper introduces a generalizable approach that uses small-scale code versions of the Qwen 2.5 model combined with Group Relative Policy Optimization (GRPO) to enable effective code generation through explicit reasoning steps, which is particularly beneficial for languages with smaller source code databases. Using Prolog as a representative use case -- given its limited online presence -- the initial model faced challenges in generating executable code. After some training steps, the model successfully produces logically consistent and syntactically accurate code by directly integrating reasoning-driven feedback into the reinforcement learning loop. Experimental evaluations using mathematical logic problem benchmarks illustrate significant improvements in reasoning quality, code accuracy, and logical correctness, underscoring the potential of this approach to benefit a wide range of programming languages lacking extensive training resources.