Statistical Validation of Column Matching in the Database Schema Evolution of the Brazilian Public School Census

TL;DR

This paper introduces a statistical method using the Kolmogorov-Smirnov test to validate and match evolving dataset schemas over 12 years of Brazil's School Census, ensuring data integration accuracy.

Contribution

It presents a novel statistical validation approach for matching dataset columns across multiple versions, addressing schema evolution challenges.

Findings

Kolmogorov-Smirnov test achieves 90% matching accuracy

Effective validation of schema changes over 12 years

Supports reliable integration of evolving datasets

Abstract

Publicly available datasets are subject to new versions, with each new version potentially reflecting changes to the data. These changes may involve adding or removing attributes, changing data types, and modifying values or their semantics. Integrating these datasets into a database poses a significant challenge: how to keep track of the evolving database schema while incorporating different versions of the data sources? This paper presents a statistical methodology to validate the integration of 12 years of open-access datasets from Brazil's School Census, with a new version of the datasets released annually by the Brazilian Ministry of Education (MEC). We employ various statistical tests to find matching attributes between datasets from a specific year and their potential equivalents in datasets from later years. The results show that by using the Kolmogorov-Smirnov test we can successfully match columns from different dataset versions in about 90% of cases.