Reconstruction of Missing Big Sensor Data

TL;DR

This paper introduces matrix and tensor-based algorithms for reconstructing missing data in IoT sensor datasets, addressing data loss caused by unreliable links or hardware failures, and demonstrates their effectiveness on real data.

Contribution

It proposes novel matrix and tensor-based methods, including the use of the alternating direction method of multipliers, for accurate missing data reconstruction in IoT sensor networks.

Findings

Effective reconstruction of missing sensor data demonstrated on real datasets.

Tensor-based methods outperform matrix-based approaches in correlated multi-sensor data.

Algorithms handle both random and consecutive missing data patterns successfully.

Abstract

With ubiquitous sensors continuously monitoring and collecting large amounts of information, there is no doubt that this is an era of big data. One of the important sources for scientific big data is the datasets collected by Internet of things (IoT). It's considered that these datesets contain highly useful and valuable information. For an IoT application to analyze big sensor data, it is necessary that the data are clean and lossless. However, due to unreliable wireless link or hardware failure in the nodes, data loss in IoT is very common. To reconstruct the missing big sensor data, firstly, we propose an algorithm based on matrix rank-minimization method. Then, we consider IoT with multiple types of sensor in each node. Accounting for possible correlations among multiple-attribute sensor data, we propose tensor-based methods to estimate missing values. Moreover, effective solutions are proposed using the alternating direction method of multipliers. Finally, we evaluate the approaches using two real sensor datasets with two missing data-patterns, i.e., random missing pattern and consecutive missing pattern. The experiments with real-world sensor data show the effectiveness of the proposed methods.