Text2Cypher Across Languages: Evaluating and Finetuning LLMs

TL;DR

This study evaluates and improves multilingual large language models for translating natural language questions into Cypher queries, revealing the impact of training data on cross-lingual performance and the benefits of multilingual finetuning.

Contribution

The paper introduces a multilingual dataset for Text2Cypher, compares foundational and finetuned models across languages, and demonstrates how multilingual finetuning enhances cross-lingual robustness.

Findings

Models perform best in English, worse in Spanish, and least in Turkish.

Prompt translation has minimal impact on evaluation metrics.

Multilingual finetuning reduces performance gaps between languages.

Abstract

Recent advances in large language models (LLMs) have enabled natural language interfaces that translate user questions into database queries, such as Text2SQL, Text2SPARQL, and Text2Cypher. While these interfaces enhance database accessibility, most research today focuses on English, with limited evaluation in other languages. This paper investigates the performance of both foundational and finetuned LLMs on the Text2Cypher task across multiple languages. We create and release a multilingual dataset by translating English questions into Spanish and Turkish while preserving the original Cypher queries, enabling fair cross-lingual comparison. Using standardized prompts and metrics, we evaluate several foundational models and observe a consistent performance pattern: highest on English, followed by Spanish, and lowest on Turkish. We attribute this to differences in training data availability and linguistic features. We also examine the impact of translating task prompts into Spanish and Turkish. Results show little to no change in evaluation metrics, suggesting prompt translation has minor impact. Furthermore, we finetune a foundational model on two datasets: one in English only, and one multilingual. Finetuning on English improves overall accuracy but widens the performance gap between languages. In contrast, multilingual finetuning narrows the gap, resulting in more balanced performance. Our findings highlight the importance for multilingual evaluation and training to build more inclusive and robust query generation systems.