Compound Schema Registry

TL;DR

This paper introduces a generalized schema evolution approach using AI and a new language to improve schema compatibility management in data streaming systems, reducing manual effort and disruptions.

Contribution

It presents a novel AI-driven framework with a specialized language for automatic schema mapping, enhancing flexibility and accuracy in schema evolution.

Findings

Improved schema mapping accuracy

Enhanced flexibility for complex schema changes

Reduced manual intervention in schema evolution

Abstract

Schema evolution is critical in managing database systems to ensure compatibility across different data versions. A schema registry typically addresses the challenges of schema evolution in real-time data streaming by managing, validating, and ensuring schema compatibility. However, current schema registries struggle with complex syntactic alterations like field renaming or type changes, which often require significant manual intervention and can disrupt service. To enhance the flexibility of schema evolution, we propose the use of generalized schema evolution (GSE) facilitated by a compound AI system. This system employs Large Language Models (LLMs) to interpret the semantics of schema changes, supporting a broader range of syntactic modifications without interrupting data streams. Our approach includes developing a task-specific language, Schema Transformation Language (STL), to generate schema mappings as an intermediate representation (IR), simplifying the integration of schema changes across different data processing platforms. Initial results indicate that this approach can improve schema mapping accuracy and efficiency, demonstrating the potential of GSE in practical applications.