Leveraging Data Preparation, HBase NoSQL Storage, and HiveQL Querying for COVID-19 Big Data Analytics Projects

TL;DR

This paper presents a detailed approach for preparing, storing, and querying COVID-19 data using HBase NoSQL and HiveQL to streamline data analysis workflows for researchers.

Contribution

It introduces specific schemas and scripts for efficient COVID-19 data formatting, storage, and querying, reducing data preparation efforts in analytics projects.

Findings

Significant reduction in data preparation time for COVID-19 analysis

Effective use of HBase and HiveQL for large-scale COVID-19 data management

Enhanced accessibility of COVID-19 data for researchers

Abstract

Epidemiologist, Scientists, Statisticians, Historians, Data engineers and Data scientists are working on finding descriptive models and theories to explain COVID-19 expansion phenomena or on building analytics predictive models for learning the apex of COVID-19 confimed cases, recovered cases, and deaths evolution curves. In CRISP-DM life cycle, 75% of time is consumed only by data preparation phase causing lot of pressions and stress on scientists and data scientists building machine learning models. This paper aims to help reducing data preparation efforts by presenting detailed schemas design and data preparation technical scripts for formatting and storing Johns Hopkins University COVID-19 daily data in HBase NoSQL data store, and enabling HiveQL COVID-19 data querying in a relational Hive SQL-like style.