Big enterprise registration data imputation: Supporting spatiotemporal analysis of industries in China

TL;DR

This paper presents a scalable HPC-based framework using Apache Spark and machine learning to impute missing and ambiguous data in large enterprise registration datasets, enabling detailed spatiotemporal industry analysis in China.

Contribution

It introduces a novel big data imputation workflow combining external data, NLP, and machine learning within HPC environments for enterprise data.

Findings

The framework is feasible and efficient for large-scale data imputation.

Imputation improves data quality for accurate spatiotemporal analysis.

The approach is scalable and adaptable to other georeferenced text data.

Abstract

Big, fine-grained enterprise registration data that includes time and location information enables us to quantitatively analyze, visualize, and understand the patterns of industries at multiple scales across time and space. However, data quality issues like incompleteness and ambiguity, hinder such analysis and application. These issues become more challenging when the volume of data is immense and constantly growing. High Performance Computing (HPC) frameworks can tackle big data computational issues, but few studies have systematically investigated imputation methods for enterprise registration data in this type of computing environment. In this paper, we propose a big data imputation workflow based on Apache Spark as well as a bare-metal computing cluster, to impute enterprise registration data. We integrated external data sources, employed Natural Language Processing (NLP), and compared several machine-learning methods to address incompleteness and ambiguity problems found in enterprise registration data. Experimental results illustrate the feasibility, efficiency, and scalability of the proposed HPC-based imputation framework, which also provides a reference for other big georeferenced text data processing. Using these imputation results, we visualize and briefly discuss the spatiotemporal distribution of industries in China, demonstrating the potential applications of such data when quality issues are resolved.