Enhancing LLM Code Generation Capabilities through Test-Driven Development and Code Interpreter

TL;DR

This paper presents a novel approach combining Test-Driven Development and Code Interpreter techniques with open-weight models to significantly improve Bengali code generation accuracy without finetuning, making advanced tools more accessible in resource-limited settings.

Contribution

It introduces a finetuning-free method that enhances Bengali code generation accuracy using TDD and CI with open-weight models, promoting democratization of AI tools.

Findings

Achieved 85% accuracy on Bengali code generation tasks.

Small models reached up to 98% accuracy, close to larger models.

Results are publicly available for validation and reproducibility.

Abstract

Over the past few years, improving LLM code generation capabilities has been a key focus in NLP research. Despite Bengali having 242 million native speakers worldwide, it receives little attention when it comes to training LLMs. More recently, various fine-tuning and augmented generation techniques have been employed to significantly enhance code generation performance. However, they require considerable expertise and resources to utilize effectively as an end user. The goal of our work is to democratize access to powerful code generation tools in resource-constrained emerging markets, enabling users to leverage them in their native language. We introduce a novel approach that combines Test-Driven Development (TDD) and Code Interpreter (CI), utilizing open-weight models, which improves the baseline accuracy for code generation with Bengali prompts and achieves an overall accuracy of 85%. Our approach requires no finetuning and proves that even the smallest models in the same family can attain up to 98% accuracy compared to the largest models. All of our results are publicly shared in GitHub for validation and reproducibility.