Exploring different approaches to customize language models for domain-specific text-to-code generation

TL;DR

This paper compares methods for customizing small language models to generate domain-specific code, finding that fine-tuning with LoRA improves accuracy while prompting methods are more cost-effective.

Contribution

It systematically evaluates three customization strategies—few-shot prompting, retrieval-augmented generation, and LoRA fine-tuning—for domain-specific code generation with small models.

Findings

LoRA fine-tuning yields higher accuracy and better domain alignment.

Prompting methods improve domain relevance but have limited impact on accuracy.

Trade-offs exist between cost, flexibility, and performance in model customization.

Abstract

Large language models (LLMs) have demonstrated strong capabilities in generating executable code from natural language descriptions. However, general-purpose models often struggle in specialized programming contexts where domain-specific libraries, APIs, or conventions must be used. Customizing smaller open-source models offers a cost-effective alternative to relying on large proprietary systems. In this work, we investigate how smaller language models can be adapted for domain-specific code generation using synthetic datasets. We construct datasets of programming exercises across three domains within the Python ecosystem: general Python programming, Scikit-learn machine learning workflows, and OpenCV-based computer vision tasks. Using these datasets, we evaluate three customization strategies: few-shot prompting, retrieval-augmented generation (RAG), and parameter-efficient fine-tuning using Low-Rank Adaptation (LoRA). Performance is evaluated using both benchmark-based metrics and similarity-based metrics that measure alignment with domain-specific code. Our results show that prompting-based approaches such as few-shot learning and RAG can improve domain relevance in a cost-effective manner, although their impact on benchmark accuracy is limited. In contrast, LoRA-based fine-tuning consistently achieves higher accuracy and stronger domain alignment across most tasks. These findings highlight practical trade-offs between flexibility, computational cost, and performance when adapting smaller language models for specialized programming tasks.