Memory-Based Multi-Processing Method For Big Data Computation

TL;DR

This paper introduces a memory-based, multi-processing approach for Big Data computation using a single-server architecture, achieving high performance and low cost compared to traditional distributed systems.

Contribution

It presents a novel non-distributed, memory-centric multi-processing method that simplifies Big Data handling while enhancing performance and reducing costs.

Findings

Outperforms conventional Big Data processing methods

Achieves high performance with low cost

Simplifies management with single-server setup

Abstract

The evolution of the Internet and computer applications have generated colossal amount of data. They are referred to as Big Data and they consist of huge volume, high velocity, and variable datasets that need to be managed at the right speed and within the right time frame to allow real-time data processing and analysis. Several Big Data solutions were developed, however they are all based on distributed computing which can be sometimes expensive to build, manage, troubleshoot, and secure. This paper proposes a novel method for processing Big Data using memory-based, multi-processing, and one-server architecture. It is memory-based because data are loaded into memory prior to start processing. It is multi-processing because it leverages the power of parallel programming using shared memory and multiple threads running over several CPUs in a concurrent fashion. It is one-server because it only requires a single server that operates in a non-distributed computing environment. The foremost advantages of the proposed method are high performance, low cost, and ease of management. The experiments conducted showed outstanding results as the proposed method outperformed other conventional methods that currently exist on the market. Further research can improve upon the proposed method so that it supports message passing between its different processes using remote procedure calls among other techniques.