Change a Bit to save Bytes: Compression for Floating Point Time-Series Data

TL;DR

This paper introduces two novel preprocessing techniques for floating point time-series data that significantly enhance compression efficiency, achieving up to 80% reduction with minimal recovery error, thereby supporting scalable IoT data management.

Contribution

The paper presents new preprocessing methods that improve compression ratios of existing IoT data compressors while enabling random access and analytics on compressed data.

Findings

Up to 80% compression reduction with 1% recovery error.

Effective with compressors supporting random access and analytics.

Enhances scalability for IoT data storage and transmission.

Abstract

The number of IoT devices is expected to continue its dramatic growth in the coming years and, with it, a growth in the amount of data to be transmitted, processed and stored. Compression techniques that support analytics directly on the compressed data could pave the way for systems to scale efficiently to these growing demands. This paper proposes two novel methods for preprocessing a stream of floating point data to improve the compression capabilities of various IoT data compressors. In particular, these techniques are shown to be helpful with recent compressors that allow for random access and analytics while maintaining good compression. Our techniques improve compression with reductions up to 80% when allowing for at most 1% of recovery error.