Statistical Learning to Operationalize a Domain Agnostic Data Quality Scoring

TL;DR

This paper presents an automated, domain-agnostic data quality scoring platform that assesses datasets using a comprehensive metric derived from multiple quality indicators, aiding data scientists in evaluating data reliability.

Contribution

The study introduces a novel automated platform that generates a data quality score, label, and report using PCA-based metrics across diverse datasets, filling a gap in practical data quality assessment.

Findings

Developed a metric with nine quality ingredients

Validated the metric using mutation testing

Demonstrated the platform's effectiveness on real datasets

Abstract

Data is expanding at an unimaginable rate, and with this development comes the responsibility of the quality of data. Data Quality refers to the relevance of the information present and helps in various operations like decision making and planning in a particular organization. Mostly data quality is measured on an ad-hoc basis, and hence none of the developed concepts provide any practical application. The current empirical study was undertaken to formulate a concrete automated data quality platform to assess the quality of incoming dataset and generate a quality label, score and comprehensive report. We utilize various datasets from healthdata.gov, opendata.nhs and Demographics and Health Surveys (DHS) Program to observe the variations in the quality score and formulate a label using Principal Component Analysis(PCA). The results of the current empirical study revealed a metric that encompasses nine quality ingredients, namely provenance, dataset characteristics, uniformity, metadata coupling, percentage of missing cells and duplicate rows, skewness of data, the ratio of inconsistencies of categorical columns, and correlation between these attributes. The study also provides an illustrative case study and validation of the metric following Mutation Testing approaches. This research study provides an automated platform which takes an incoming dataset and metadata to provide the DQ score, report and label. The results of this study would be useful to data scientists as the value of this quality label would instill confidence before deploying the data for his/her respective practical application.