Share the Tensor Tea: How Databases can Leverage the Machine Learning Ecosystem

TL;DR

Tensor Query Processor (TQP) enables databases to seamlessly integrate with machine learning ecosystems by compiling relational queries into tensor programs, leveraging ML runtimes and hardware for efficient, end-to-end query acceleration.

Contribution

Introduces TQP, a novel query processor that compiles relational operators into tensor programs, bridging databases and machine learning tools.

Findings

TQP integrates with ML tools like Pandas and Tensorboard.

TQP supports multiple hardware backends including CPU and GPU.

TQP achieves performance comparable or superior to specialized query processors.

Abstract

We demonstrate Tensor Query Processor (TQP): a query processor that automatically compiles relational operators into tensor programs. By leveraging tensor runtimes such as PyTorch, TQP is able to: (1) integrate with ML tools (e.g., Pandas for data ingestion, Tensorboard for visualization); (2) target different hardware (e.g., CPU, GPU) and software (e.g., browser) backends; and (3) end-to-end accelerate queries containing both relational and ML operators. TQP is generic enough to support the TPC-H benchmark, and it provides performance that is comparable to, and often better than, that of specialized CPU and GPU query processors.