Compiler-Guided Inference-Time Adaptation: Improving GPT-5 Programming Performance in Idris

TL;DR

This paper demonstrates that GPT-5's programming performance in a low-resource language like Idris can be significantly improved through compiler-guided, iterative feedback prompting, achieving near-complete problem-solving success.

Contribution

It introduces a compiler-guided, iterative prompting method that enhances GPT-5's ability to learn and perform in low-resource programming languages.

Findings

Zero-shot GPT-5 solves 22/56 Idris exercises.

Iterative, error-guided prompting increases success to 54/56.

Structured feedback is crucial for improving model performance in low-resource settings.

Abstract

GPT-5, a state of the art large language model from OpenAI, demonstrates strong performance in widely used programming languages such as Python, C++, and Java; however, its ability to operate in low resource or less commonly used languages remains underexplored. This work investigates whether GPT-5 can effectively acquire proficiency in an unfamiliar functional programming language, Idris, through iterative, feedback driven prompting. We first establish a baseline showing that with zero shot prompting the model solves only 22 out of 56 Idris exercises using the platform Exercism, substantially underperforming relative to higher resource languages (45 out of 50 in Python and 35 out of 47 in Erlang). We then evaluate several refinement strategies, including iterative prompting based on platform feedback, augmenting prompts with documentation and error classification guides, and iterative prompting using local compilation errors and failed test cases. Among these approaches, incorporating local compilation errors yields the most substantial improvements. Using this structured, error guided refinement loop, GPT-5 performance increased to an impressive 54 solved problems out of 56. These results suggest that while large language models may initially struggle in low resource settings, structured compiler level feedback can play a critical role in unlocking their capabilities.