Evaluating Retrieval-Augmented Generation Variants for Natural Language-Based SQL and API Call Generation

TL;DR

This paper evaluates retrieval-augmented generation variants for translating natural language into SQL and API calls in enterprise systems, highlighting the importance of retrieval strategies for accurate code generation.

Contribution

It provides a comprehensive comparison of RAG, Self-RAG, and CoRAG in enterprise contexts, demonstrating the effectiveness of retrieval policies in hybrid documentation scenarios.

Findings

Retrieval significantly improves code accuracy, with up to 79.30% execution accuracy.

CoRAG outperforms other variants in hybrid documentation settings.

Iterative query decomposition enhances retrieval effectiveness over other methods.

Abstract

Enterprise systems increasingly require natural language interfaces that can translate user requests into structured operations such as SQL queries and REST API calls. While large language models (LLMs) show promise for code generation [Chen et al., 2021; Huynh and Lin, 2025], their effectiveness in domain-specific enterprise contexts remains underexplored, particularly when both retrieval and modification tasks must be handled jointly. This paper presents a comprehensive evaluation of three retrieval-augmented generation (RAG) variants [Lewis et al., 2021] -- standard RAG, Self-RAG [Asai et al., 2024], and CoRAG [Wang et al., 2025] -- across SQL query generation, REST API call generation, and a combined task requiring dynamic task classification. Using SAP Transactional Banking as a realistic enterprise use case, we construct a novel test dataset covering both modalities and evaluate 18 experimental configurations under database-only, API-only, and hybrid documentation contexts. Results demonstrate that RAG is essential: Without retrieval, exact match accuracy is 0% across all tasks, whereas retrieval yields substantial gains in execution accuracy (up to 79.30%) and component match accuracy (up to 78.86%). Critically, CoRAG proves most robust in hybrid documentation settings, achieving statistically significant improvements in the combined task (10.29% exact match vs. 7.45% for standard RAG), driven primarily by superior SQL generation performance (15.32% vs. 11.56%). Our findings establish retrieval-policy design as a key determinant of production-grade natural language interfaces, showing that iterative query decomposition outperforms both top-k retrieval and binary relevance filtering under documentation heterogeneity.