Prompt Engineering Techniques for Context-dependent Text-to-SQL in Arabic

TL;DR

This paper introduces Ar-SParC, the first Arabic cross-domain, context-dependent text-to-SQL dataset, and evaluates prompt engineering techniques and a novel GAT corrector to improve model performance in Arabic SQL generation.

Contribution

It presents Ar-SParC dataset for Arabic, explores prompt engineering with large language models, and proposes GAT corrector to enhance accuracy in Arabic text-to-SQL tasks.

Findings

GAT corrector improved execution and interaction accuracy by around 1.9%.

40 experiments conducted with GPT-3.5-turbo and GPT-4.5-turbo.

First Arabic dataset for cross-domain, context-dependent text-to-SQL.

Abstract

In recent years, the task of cross-domain, context-dependent text-to-SQL has received significant attention. Enables users with no prior knowledge of SQL to have a conversation with databases using natural language. However, most of the available datasets and research have been conducted in English, along with some work in Chinese. To this date, no effort has been made to address this task in the Arabic language. In this paper, we introduce Ar-SParC, the first Arabic cross-domain, context-dependent text-to-SQL dataset. The dataset consists of 3,450 sequences of interrelated questions, each sequence containing an average of approximately three questions, which results in a total of 10225 questions along with their corresponding SQL queries. We conducted 40 experiments on the Ar-SParC dataset using two large language models, GPT-3.5-turbo and GPT-4.5-turbo, applying 10 different prompt engineering techniques, including four question representation methods and six in-context learning techniques. Furthermore, we developed a novel approach named GAT corrector, which enhanced the performance across all 40 experiments, yielding an average improvement of 1.9% in execution accuracy (EX) and 1.9% in interaction accuracy (IX) under zero-shot settings, and an average increase of 1.72% EX and 0.92% IX under in-context learning settings. Finally, we conducted an ablation study with two more experiments to explain why the GAT corrector outperformed the previous GAT verifier technique, particularly for the Arabic language.