Bridging BAD Islands: Declarative Data Sharing at Scale

TL;DR

This paper introduces a declarative mechanism for scalable Big Data sharing across organizations, simplifying development and management by building on the existing BAD system and demonstrating its effectiveness through a prototype.

Contribution

It presents a novel declarative approach for Big Data sharing at scale, extending the BAD system to facilitate easier and scalable data sharing services.

Findings

Prototype system demonstrates scalable data sharing capabilities.

Experimental results show improved ease of creating data sharing services.

The approach reduces development effort for cross-organizational Big Data sharing.

Abstract

In many Big Data applications today, information needs to be actively shared between systems managed by different organizations. To enable sharing Big Data at scale, developers would have to create dedicated server programs and glue together multiple Big Data systems for scalability. Developing and managing such glued data sharing services requires a significant amount of work from developers. In our prior work, we developed a Big Active Data (BAD) system for enabling Big Data subscriptions and analytics with millions of subscribers. Based on that, we introduce a new mechanism for enabling the sharing of Big Data at scale declaratively so that developers can easily create and provide data sharing services using declarative statements and can benefit from an underlying scalable infrastructure. We show our implementation on top of the BAD system, explain the data sharing data flow among multiple systems, and present a prototype system with experimental results.