The BigDAWG Polystore System and Architecture

TL;DR

BigDAWG is a polystore system designed to manage complex, heterogeneous datasets across multiple database engines, improving performance and flexibility for diverse data types like medical data.

Contribution

The paper introduces the BigDAWG polystore architecture, enabling seamless integration of different database systems for complex, multi-model data management.

Findings

Prototype applied to medical dataset validates polystore concepts

Initial performance results show promising efficiency

Supports diverse data models with a unified interface

Abstract

Organizations are often faced with the challenge of providing data management solutions for large, heterogenous datasets that may have different underlying data and programming models. For example, a medical dataset may have unstructured text, relational data, time series waveforms and imagery. Trying to fit such datasets in a single data management system can have adverse performance and efficiency effects. As a part of the Intel Science and Technology Center on Big Data, we are developing a polystore system designed for such problems. BigDAWG (short for the Big Data Analytics Working Group) is a polystore system designed to work on complex problems that naturally span across different processing or storage engines. BigDAWG provides an architecture that supports diverse database systems working with different data models, support for the competing notions of location transparency and semantic completeness via islands and a middleware that provides a uniform multi--island interface. Initial results from a prototype of the BigDAWG system applied to a medical dataset validate polystore concepts. In this article, we will describe polystore databases, the current BigDAWG architecture and its application on the MIMIC II medical dataset, initial performance results and our future development plans.