Energy Consumption of Dataframe Libraries for End-to-End Deep Learning Pipelines:A Comparative Analysis

TL;DR

This study compares Pandas, Polars, and Dask in deep learning pipelines, focusing on runtime, memory, disk, and energy efficiency during data handling for GPU workloads.

Contribution

It provides a comprehensive analysis of how popular data libraries perform in end-to-end deep learning workflows, highlighting their energy consumption and efficiency.

Findings

Polars often outperforms Pandas and Dask in speed and energy efficiency.

Data loading and preprocessing significantly impact overall energy consumption.

GPU and CPU energy use varies notably across libraries and datasets.

Abstract

This paper presents a detailed comparative analysis of the performance of three major Python data manipulation libraries - Pandas, Polars, and Dask - specifically when embedded within complete deep learning (DL) training and inference pipelines. The research bridges a gap in existing literature by studying how these libraries interact with substantial GPU workloads during critical phases like data loading, preprocessing, and batch feeding. The authors measured key performance indicators including runtime, memory usage, disk usage, and energy consumption (both CPU and GPU) across various machine learning models and datasets.