TableQuery: Querying tabular data with natural language

TL;DR

TableQuery introduces a deep learning-based approach that converts natural language questions into structured database queries, enabling efficient querying of large or live data without loading entire tables into memory.

Contribution

It leverages pre-trained question answering models to translate natural language into structured queries, avoiding the need for re-training and handling large or live datasets efficiently.

Findings

Eliminates memory limitations by converting queries without loading entire tables.

Supports real-time data querying from live databases and spreadsheets.

Can easily update models without retraining the entire system.

Abstract

This paper presents TableQuery, a novel tool for querying tabular data using deep learning models pre-trained to answer questions on free text. Existing deep learning methods for question answering on tabular data have various limitations, such as having to feed the entire table as input into a neural network model, making them unsuitable for most real-world applications. Since real-world data might contain millions of rows, it may not entirely fit into the memory. Moreover, data could be stored in live databases, which are updated in real-time, and it is impractical to serialize an entire database to a neural network-friendly format each time it is updated. In TableQuery, we use deep learning models pre-trained for question answering on free text to convert natural language queries to structured queries, which can be run against a database or a spreadsheet. This method eliminates the need for fitting the entire data into memory as well as serializing databases. Furthermore, deep learning models pre-trained for question answering on free text are readily available on platforms such as HuggingFace Model Hub (7). TableQuery does not require re-training; when a newly trained model for question answering with better performance is available, it can replace the existing model in TableQuery.