Missing Data Estimation in High-Dimensional Datasets: A Swarm Intelligence-Deep Neural Network Approach

TL;DR

This paper introduces a novel approach combining deep learning and swarm intelligence to accurately estimate missing data in high-dimensional datasets, addressing various missing data mechanisms.

Contribution

It proposes a new hybrid methodology leveraging deep feature extraction and swarm optimization for missing data imputation in complex datasets.

Findings

Effective estimation of missing data under different missing mechanisms.

Improved accuracy over traditional imputation methods.

Potential for application in high-dimensional data analysis.

Abstract

In this paper, we examine the problem of missing data in high-dimensional datasets by taking into consideration the Missing Completely at Random and Missing at Random mechanisms, as well as theArbitrary missing pattern. Additionally, this paper employs a methodology based on Deep Learning and Swarm Intelligence algorithms in order to provide reliable estimates for missing data. The deep learning technique is used to extract features from the input data via an unsupervised learning approach by modeling the data distribution based on the input. This deep learning technique is then used as part of the objective function for the swarm intelligence technique in order to estimate the missing data after a supervised fine-tuning phase by minimizing an error function based on the interrelationship and correlation between features in the dataset. The investigated methodology in this paper therefore has longer running times, however, the promising potential outcomes justify the trade-off. Also, basic knowledge of statistics is presumed.