Can Language Models Analyze Data? Evaluating Large Language Models for Question Answering over Datasets

TL;DR

This study evaluates large language models' ability to answer questions over datasets and generate SQL queries, comparing performance across different models and prompting strategies.

Contribution

It provides a comprehensive analysis of LLMs' effectiveness in data question answering and SQL generation, highlighting their strengths and limitations.

Findings

Large LLMs perform strongly in dataset question answering.

Smaller models are less effective but more resource-efficient.

Prompting strategies significantly impact model performance.

Abstract

This paper investigates the effectiveness of large language models (LLMs) in answering questions over datasets. We examine their performance in two scenarios: (a) directly answering questions given a dataset file as input, and (b) generating SQL queries to answer questions given the schema of a relational database. We also evaluate the impact of different prompting strategies on model performance. The study includes both state-of-the-art LLMs and smaller language models that require fewer resources and operate at lower computational and financial cost. Experiments are conducted on two datasets containing questions of varying difficulty. The results demonstrate the strong performance of large LLMs, while highlighting the limitations of smaller, more cost-efficient models. These findings contribute to a better understanding of how LLMs can be utilized in data analytics tasks and their associated limitations.