Detecting Quality Problems in Data Models by Clustering Heterogeneous Data Values

TL;DR

This paper presents a bottom-up clustering approach to identify data quality issues caused by heterogeneity in data values, aiding domain experts in understanding and improving data models.

Contribution

It introduces a novel method for detecting data model quality problems through clustering heterogeneous data values, supporting domain expert analysis.

Findings

Effective in revealing data heterogeneity in practice

Supports domain experts in identifying data quality issues

Validated on cultural heritage data

Abstract

Data is of high quality if it is fit for its intended use. The quality of data is influenced by the underlying data model and its quality. One major quality problem is the heterogeneity of data as quality aspects such as understandability and interoperability are impaired. This heterogeneity may be caused by quality problems in the data model. Data heterogeneity can occur in particular when the information given is not structured enough and just captured in data values, often due to missing or non-suitable structure in the underlying data model. We propose a bottom-up approach to detecting quality problems in data models that manifest in heterogeneous data values. It supports an explorative analysis of the existing data and can be configured by domain experts according to their domain knowledge. All values of a selected data field are clustered by syntactic similarity. Thereby an overview of the data values' diversity in syntax is provided. It shall help domain experts to understand how the data model is used in practice and to derive potential quality problems of the data model. We outline a proof-of-concept implementation and evaluate our approach using cultural heritage data.