Development of Data Evaluation Benchmark for Data Wrangling Recommendation System

TL;DR

This paper introduces CoWrangler, a data-wrangling recommendation system, developed through analyzing thousands of Kaggle notebooks to understand common data processing strategies and improve decision-making for data tasks.

Contribution

The paper presents a new benchmark for evaluating data wrangling recommendations based on extensive analysis of real-world user data from Kaggle notebooks.

Findings

Insights into common data processing strategies

Understanding of how dataset quality affects wrangling operations

Potential for expanding dataset sources in future work

Abstract

CoWrangler is a data-wrangling recommender system designed to streamline data processing tasks. Recognizing that data processing is often time-consuming and complex for novice users, we aim to simplify the decision-making process regarding the most effective subsequent data operation. By analyzing over 10,000 Kaggle notebooks spanning approximately 1,000 datasets, we derive insights into common data processing strategies employed by users across various tasks. This analysis helps us understand how dataset quality influences wrangling operations, informing our ongoing efforts to possibly expand our dataset sources in the future.