d-DQIVAR: Data-centric Visual Analytics and Reasoning for Data Quality Improvement

TL;DR

d-DQIVAR is a visual analytics system that combines data-driven and process-driven techniques to improve data quality specifically for enhancing machine learning model performance, addressing limitations of traditional batch preprocessing methods.

Contribution

The paper introduces d-DQIVAR, a novel visual analytics tool that integrates data and process-driven strategies for genuine data quality improvement in ML workflows.

Findings

Effective identification of data quality issues using visual analytics.

Improved ML model performance through targeted data quality strategies.

User studies demonstrate practical utility of the system.

Abstract

Approaches to enhancing data quality (DQ) are classified into two main categories: data- and process-driven. However, prior research has predominantly utilized batch data preprocessing within the data-driven framework, which often proves insufficient for optimizing machine learning (ML) model performance and frequently leads to distortions in data characteristics. Existing studies have primarily focused on data preprocessing rather than genuine data quality improvement (DQI). In this paper, we introduce d-DQIVAR, a novel visual analytics system designed to facilitate DQI strategies aimed at improving ML model performance. Our system integrates visual analytics techniques that leverage both data-driven and process-driven approaches. Data-driven techniques tackle DQ issues such as imputation, outlier detection, deletion, format standardization, removal of duplicate records, and feature selection. Process-driven strategies encompass evaluating DQ and DQI procedures by considering DQ dimensions and ML model performance and applying the Kolmogorov-Smirnov test. We illustrate how our system empowers users to harness expert and domain knowledge effectively within a practical workflow through case studies, evaluations, and user studies.