JSONoid: Monoid-based Enrichment for Configurable and Scalable Data-Driven Schema Discovery

TL;DR

JSONoid is a distributed schema discovery method for JSON data that uses monoid data structures to efficiently generate schemas and enrich them with valuable metadata, scalable to large datasets.

Contribution

It introduces a monoid-based approach for scalable, distributed schema discovery that also provides enriched metadata about data values.

Findings

Performs comparably to existing distributed schema discovery methods

Enriches schemas with additional data value metadata

Scales linearly with dataset size

Abstract

Schema discovery is an important aspect to working with data in formats such as JSON. Unlike relational databases, JSON data sets often do not have associated structural information. Consumers of such datasets are often left to browse through data in an attempt to observe commonalities in structure across documents to construct suitable code for data processing. However, this process is time-consuming and error-prone. Existing distributed approaches to mining schemas present a significant usability advantage as they provide useful metadata for large data sources. However, depending on the data source, ad hoc queries for estimating other properties to help with crafting an efficient data pipeline can be expensive. We propose JSONoid, a distributed schema discovery process augmented with additional metadata in the form of monoid data structures that are easily maintainable in a distributed setting. JSONoid subsumes several existing approaches to distributed schema discovery with similar performance. Our approach also adds significant useful additional information about data values to discovered schemas with linear scalability.