Multistage Large Segment Imputation Framework Based on Deep Learning and Statistic Metrics

TL;DR

This paper introduces a multistage deep learning framework for sensor data missing value imputation, utilizing a novel mixture index and adaptive strategies to improve accuracy, especially for large segments.

Contribution

It proposes a new multistage imputation framework with a mixture statistical index and adaptive data handling, addressing data distribution and period variability in sensor data.

Findings

Multistage strategy improves imputation accuracy for large data segments.

Mixture index outperforms traditional mean squared error in evaluation.

Framework shows superior performance across different sensor data types.

Abstract

Missing value is a very common and unavoidable problem in sensors, and researchers have made numerous attempts for missing value imputation, particularly in deep learning models. However, for real sensor data, the specific data distribution and data periods are rarely considered, making it difficult to choose the appropriate evaluation indexes and models for different sensors. To address this issue, this study proposes a multistage imputation framework based on deep learning with adaptability for missing value imputation. The model presents a mixture measurement index of low- and higher-order statistics for data distribution and a new perspective on data imputation performance metrics, which is more adaptive and effective than the traditional mean squared error. A multistage imputation strategy and dynamic data length are introduced into the imputation process for data periods. Experimental results on different types of sensor data show that the multistage imputation strategy and the mixture index are superior and that the effect of missing value imputation has been improved to some extent, particularly for the large segment imputation problem. The codes and experimental results have been uploaded to GitHub.