Anomaly Detection and Redundancy Elimination of Big Sensor Data in Internet of Things

TL;DR

This paper presents a comprehensive framework for preprocessing big sensor data in IoT, focusing on anomaly detection using statistical and Bayesian methods, and redundancy elimination with static and dynamic Bayesian networks, validated on real datasets.

Contribution

It introduces novel algorithms for anomaly detection and redundancy elimination tailored for large-scale IoT sensor data, enhancing data quality and processing efficiency.

Findings

Proposed methods effectively detect anomalies in sensor data.

Redundancy elimination algorithms improve data processing efficiency.

Validated on real-world datasets with positive results.

Abstract

In the era of big data and Internet of things, massive sensor data are gathered with Internet of things. Quantity of data captured by sensor networks are considered to contain highly useful and valuable information. However, for a variety of reasons, received sensor data often appear abnormal. Therefore, effective anomaly detection methods are required to guarantee the quality of data collected by those sensor nodes. Since sensor data are usually correlated in time and space, not all the gathered data are valuable for further data processing and analysis. Preprocessing is necessary for eliminating the redundancy in gathered massive sensor data. In this paper, the proposed work defines a sensor data preprocessing framework. It is mainly composed of two parts, i.e., sensor data anomaly detection and sensor data redundancy elimination. In the first part, methods based on principal statistic analysis and Bayesian network is proposed for sensor data anomaly detection. Then, approaches based on static Bayesian network (SBN) and dynamic Bayesian networks (DBNs) are proposed for sensor data redundancy elimination. Static sensor data redundancy detection algorithm (SSDRDA) for eliminating redundant data in static datasets and real-time sensor data redundancy detection algorithm (RSDRDA) for eliminating redundant sensor data in real-time are proposed. The efficiency and effectiveness of the proposed methods are validated using real-world gathered sensor datasets.