Messy Code Makes Managing ML Pipelines Difficult? Just Let LLMs Rewrite the Code!

TL;DR

This paper proposes using large language models to automatically rewrite messy data science code into clean, declarative ML pipelines, addressing the gap between real-world code and the abstractions needed for better management and compliance.

Contribution

It introduces a novel approach leveraging LLMs to transform imperatively written data science code into declarative pipelines, facilitating better pipeline management without changing existing development practices.

Findings

LLMs can identify and convert hand-coded joins into dataframe joins.

LLMs can generate declarative feature encoders from NumPy code.

The prototype Lester demonstrates effective code rewriting for compliance tasks.

Abstract

Machine learning (ML) applications that learn from data are increasingly used to automate impactful decisions. Unfortunately, these applications often fall short of adequately managing critical data and complying with upcoming regulations. A technical reason for the persistence of these issues is that the data pipelines in common ML libraries and cloud services lack fundamental declarative, data-centric abstractions. Recent research has shown how such abstractions enable techniques like provenance tracking and automatic inspection to help manage ML pipelines. Unfortunately, these approaches lack adoption in the real world because they require clean ML pipeline code written with declarative APIs, instead of the messy imperative Python code that data scientists typically write for data preparation. We argue that it is unrealistic to expect data scientists to change their established development practices. Instead, we propose to circumvent this "code abstraction gap" by leveraging the code generation capabilities of large language models (LLMs). Our idea is to rewrite messy data science code to a custom-tailored declarative pipeline abstraction, which we implement as a proof-of-concept in our prototype Lester. We detail its application for a challenging compliance management example involving "incremental view maintenance" of deployed ML pipelines. The code rewrites for our running example show the potential of LLMs to make messy data science code declarative, e.g., by identifying hand-coded joins in Python and turning them into joins on dataframes, or by generating declarative feature encoders from NumPy code.