The Impact of Data Compression in Real-Time and Historical Data Acquisition Systems on the Accuracy of Analytical Solutions

TL;DR

This paper investigates how data compression in industrial IoT systems affects the accuracy of analytics, highlighting trade-offs and proposing best practices for balancing efficiency and reliability.

Contribution

It provides a comprehensive analysis of compression impacts on analytical accuracy, combining theoretical, simulated, and empirical methods to guide optimal data management strategies.

Findings

Excessive compression can obscure critical data patterns.

Compression may skew statistical analysis results.

Optimal compression balances efficiency and analytical integrity.

Abstract

In industrial and IoT environments, massive amounts of real-time and historical process data are continuously generated and archived. With sensors and devices capturing every operational detail, the volume of time-series data has become a critical challenge for storage and processing systems. Efficient data management is essential to ensure scalability, cost-effectiveness, and timely analytics. To minimize storage expenses and optimize performance, data compression algorithms are frequently utilized in data historians and acquisition systems. However, compression comes with trade-offs that may compromise the accuracy and reliability of engineering analytics that depend on this compressed data. Understanding these trade-offs is essential for developing data strategies that support both operational efficiency and accurate, reliable analytics. This paper assesses the relation of common compression mechanisms used in real-time and historical data systems and the accuracy of analytical solutions, including statistical analysis, anomaly detection, and machine learning models. Through theoretical analysis, simulated signal compression, and empirical assessment, we illustrate that excessive compression can lose critical patterns, skew statistical measures, and diminish predictive accuracy. The study suggests optimum methods and best practices for striking a compromise between analytical integrity and compression efficiency.