A Comparative Study of DSL Code Generation: Fine-Tuning vs. Optimized Retrieval Augmentation

TL;DR

This study compares fine-tuning and retrieval-augmented methods for generating domain-specific language code, showing RAG can match fine-tuning in quality and better handle new APIs despite some syntax errors.

Contribution

The paper presents optimized RAG strategies for DSL code generation and compares them with fine-tuning, highlighting RAG's effectiveness for unseen APIs and ongoing challenges.

Findings

Fine-tuned model scored highest on code similarity.

RAG-based approach achieved parity in similarity metrics.

Both models frequently produced syntax errors, with RAG slightly better.

Abstract

Natural Language to Code Generation has made significant progress in recent years with the advent of Large Language Models(LLMs). While generation for general-purpose languages like C, C++, and Python has improved significantly, LLMs struggle with custom function names in Domain Specific Languages or DSLs. This leads to higher hallucination rates and syntax errors, specially for DSLs having a high number of custom function names. Additionally, constant updates to function names add to the challenge as LLMs need to stay up-to-date. In this paper, we present optimizations for using Retrieval Augmented Generation (or RAG) with LLMs for DSL generation along with an ablation study comparing these strategies. We generated a train as well as test dataset with a DSL to represent automation tasks across roughly 700 APIs in public domain. We used the training dataset to fine-tune a Codex model for this DSL. Our results showed that the fine-tuned model scored the best on code similarity metric. With our RAG optimizations, we achieved parity for similarity metric. The compilation rate, however, showed that both the models still got the syntax wrong many times, with RAG-based method being 2 pts better. Conversely, hallucination rate for RAG model lagged by 1 pt for API names and by 2 pts for API parameter keys. We conclude that an optimized RAG model can match the quality of fine-tuned models and offer advantages for new, unseen APIs.