Can LLMs substitute SQL? Comparing Resource Utilization of Querying LLMs versus Traditional Relational Databases

TL;DR

This paper compares the resource use and accuracy of large language models versus traditional SQL databases for interpreting natural language queries, finding LLMs are resource-intensive and environmentally costly.

Contribution

It provides an empirical evaluation of multiple LLMs' resource consumption and accuracy in database querying, highlighting their inefficiency compared to traditional databases.

Findings

LLMs consume significant energy even at small sizes

Using LLMs for database queries is environmentally unfriendly

Traditional SQL remains more resource-efficient

Abstract

Large Language Models (LLMs) can automate or substitute different types of tasks in the software engineering process. This study evaluates the resource utilization and accuracy of LLM in interpreting and executing natural language queries against traditional SQL within relational database management systems. We empirically examine the resource utilization and accuracy of nine LLMs varying from 7 to 34 Billion parameters, including Llama2 7B, Llama2 13B, Mistral, Mixtral, Optimus-7B, SUS-chat-34B, platypus-yi-34b, NeuralHermes-2.5-Mistral-7B and Starling-LM-7B-alpha, using a small transaction dataset. Our findings indicate that using LLMs for database queries incurs significant energy overhead (even small and quantized models), making it an environmentally unfriendly approach. Therefore, we advise against replacing relational databases with LLMs due to their substantial resource utilization.