Statistical Validation of Column Matching in the Database Schema Evolution of the Brazilian Public School Census
Muriki G. Yamanaka, Diogo H. de Almeida, Paulo R. Lisboa de Almeida,, Simone Dominico, Leticia M. Peres, Marcos S. Sunye, Eduardo C. de Almeida

TL;DR
This paper introduces a statistical method using the Kolmogorov-Smirnov test to validate and match evolving dataset schemas over 12 years of Brazil's School Census, ensuring data integration accuracy.
Contribution
It presents a novel statistical validation approach for matching dataset columns across multiple versions, addressing schema evolution challenges.
Findings
Kolmogorov-Smirnov test achieves 90% matching accuracy
Effective validation of schema changes over 12 years
Supports reliable integration of evolving datasets
Abstract
Publicly available datasets are subject to new versions, with each new version potentially reflecting changes to the data. These changes may involve adding or removing attributes, changing data types, and modifying values or their semantics. Integrating these datasets into a database poses a significant challenge: how to keep track of the evolving database schema while incorporating different versions of the data sources? This paper presents a statistical methodology to validate the integration of 12 years of open-access datasets from Brazil's School Census, with a new version of the datasets released annually by the Brazilian Ministry of Education (MEC). We employ various statistical tests to find matching attributes between datasets from a specific year and their potential equivalents in datasets from later years. The results show that by using the Kolmogorov-Smirnov test we can…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCensus and Population Estimation · Data Quality and Management
