PyTond: Efficient Python Data Science on the Shoulders of Databases

TL;DR

PyTond enhances Python data science workflows by integrating them with database engines, leveraging IR-level optimizations for improved performance and scalability in handling big data workloads.

Contribution

Introduces PyTond, a novel approach that pushes data science processing into database engines using TondIR for comprehensive workload capture and IR-level SQL optimization.

Findings

Significant performance improvements over Python and other tools.

Effective handling of diverse data science workloads.

Enhanced scalability for big data processing.

Abstract

Python data science libraries such as Pandas and NumPy have recently gained immense popularity. Although these libraries are feature-rich and easy to use, their scalability limitations require more robust computational resources. In this paper, we present PyTond, an efficient approach to push the processing of data science workloads down into the database engines that are already known for their big data handling capabilities. Compared to the previous work, by introducing TondIR, our approach can capture a more comprehensive set of workloads and data layouts. Moreover, by doing IR-level optimizations, we generate better SQL code that improves the query processing by the underlying database engine. Our evaluation results show promising performance improvement compared to Python and other alternatives for diverse data science workloads.