INGESTBASE: A Declarative Data Ingestion System

TL;DR

INGESTBASE is a declarative system that streamlines big data ingestion by allowing systematic planning, optimization, and fault-tolerant execution of data preprocessing directly during ingestion, improving efficiency and flexibility.

Contribution

It introduces a declarative ingestion language and optimizer for systematic, efficient, and flexible data ingestion in big data systems, integrating preprocessing into ingestion workflows.

Findings

Supports various ingestion techniques with low overhead

Achieves up to 6x performance improvement over traditional post-ingestion processing

Provides efficient, fault-tolerant distributed ingestion and data access

Abstract

Big data applications have fast arriving data that must be quickly ingested. At the same time, they have specific needs to preprocess and transform the data before it could be put to use. The current practice is to do these preparatory transformations once the data is already ingested, however, this is expensive to run and cumbersome to manage. As a result, there is a need to push data preprocessing down to the ingestion itself. In this paper, we present a declarative data ingestion system, called INGESTBASE, to allow application developers to plan and specify their data ingestion logic in a more systematic manner. We introduce the notion of ingestions plans, analogous to query plans, and present a declarative ingestion language to help developers easily build sophisticated ingestion plans. INGESTBASE provides an extensible ingestion optimizer to rewrite and optimize ingestion plans by applying rules such as operator reordering and pipelining. Finally, the INGESTBASE runtime engine runs the optimized ingestion plan in a distributed and fault-tolerant manner. Later, at query processing time, INGESTBASE supports ingestion-aware data access and interfaces with upstream query processors, such as Hadoop MapReduce and Spark, to post- process the ingested data. We demonstrate through a number of experiments that INGESTBASE: (i) is flexible enough to express a variety of ingestion techniques, (ii) incurs a low ingestion overhead, (iii) provides efficient access to the ingested data, and (iv) has much better performance, up to 6 times, than preparing data as an afterthought, via a query processor.