Oseba: Optimization for Selective Bulk Analysis in Big Data Processing

TL;DR

Oseba introduces an in-memory super index to optimize selective bulk analysis in big data, significantly reducing memory usage and computation time compared to existing frameworks like Spark.

Contribution

The paper presents Oseba, a novel method that enhances data organization and lookup efficiency for selective bulk analysis in large-scale data processing.

Findings

Oseba reduces memory consumption compared to default frameworks.

Oseba accelerates lookup times for selective analysis.

Oseba improves overall processing efficiency in big data environments.

Abstract

Selective bulk analyses, such as statistical learning on temporal/spatial data, are fundamental to a wide range of contemporary data analysis. However, with the increasingly larger data-sets, such as weather data and marketing transactions, the data organization/access becomes more challenging in selective bulk data processing with the use of current big data processing frameworks such as Spark or keyvalue stores. In this paper, we propose a method to optimize selective bulk analysis in big data processing and referred to as Oseba. Oseba maintains a super index for the data organization in memory to support fast lookup through targeting the data involved with each selective analysis program. Oseba is able to save memory as well as computation in comparison to the default data processing frameworks.