Measuring Swampiness: Quantifying Chaos in Large Heterogeneous Data Repositories

TL;DR

This paper introduces an automated clustering method to quantify the organization of large, heterogeneous data repositories, providing a novel 'cleanliness' score to assess chaos and improve data management.

Contribution

It presents a parallel clustering pipeline that processes diverse file types and introduces a new 'cleanliness' metric validated on synthetic and real datasets.

Findings

The 'cleanliness' score correlates well with data organization levels.

The method outperforms existing measures in consistency.

It effectively handles heterogeneous data types.

Abstract

As scientific data repositories and filesystems grow in size and complexity, they become increasingly disorganized. The coupling of massive quantities of data with poor organization makes it challenging for scientists to locate and utilize relevant data, thus slowing the process of analyzing data of interest. To address these issues, we explore an automated clustering approach for quantifying the organization of data repositories. Our parallel pipeline processes heterogeneous filetypes (e.g., text and tabular data), automatically clusters files based on content and metadata similarities, and computes a novel "cleanliness" score from the resulting clustering. We demonstrate the generation and accuracy of our cleanliness measure using both synthetic and real datasets, and conclude that it is more consistent than other potential cleanliness measures.