Efficient Dataframe Systems: Lazy Fat Pandas on a Diet

TL;DR

This paper introduces a framework that optimizes Pandas code for large datasets by enabling lazy evaluation and backend flexibility, improving performance without rewriting code.

Contribution

The authors present a system that allows plain Pandas code to be optimized with minimal changes, supporting multiple backends like Pandas, Dask, and Modin.

Findings

Significant performance improvements on various programs.

Seamless integration with existing Pandas code.

Flexible backend selection based on data size.

Abstract

Pandas is widely used for data science applications, but users often run into problems when datasets are larger than memory. There are several frameworks based on lazy evaluation that handle large datasets, but the programs have to be rewritten to suit the framework, and the presence of multiple frameworks complicates the life of a programmer. In this paper we present a framework that allows programmers to code in plain Pandas; with just two lines of code changed by the user, our system optimizes the program using a combination of just-in-time static analysis, and runtime optimization based on a lazy dataframe wrapper framework. Moreover, our system allows the programmer to choose the backend. It works seamlessly with Pandas, Dask, and Modin, allowing the choice of the best-suited backend for an application based on factors such as data size. Performance results on a variety of programs show the benefits of our framework.