Can the Multi-Incoming Smart Meter Compressed Streams be Re-Compressed?

TL;DR

This paper investigates re-compressing smart meter data streams, proposing a novel unsupervised learning-based technique that significantly improves compression efficiency by reducing entropy and increasing redundancy.

Contribution

It introduces a new re-compression method using clustering, derivative, rotation, and entropy coding to enhance existing compressed smart meter streams.

Findings

Entropy reduced by nearly 50%

Compression ratio improved up to 50%

Method effective on real-world data

Abstract

Smart meters have currently attracted attention because of their high efficiency and throughput performance. They transmit a massive volume of continuously collected waveform readings (e.g. monitoring). Although many compression models are proposed, the unexpected size of these compressed streams required endless storage and management space which poses a unique challenge. Therefore, this paper explores the question of can the compressed smart meter readings be re-compressed? We first investigate the applicability of re-applying general compression algorithms directly on compressed streams. The results were poor due to the lack of redundancy. We further propose a novel technique to enhance the theoretical entropy and exploit that to re-compress. This is successfully achieved by using unsupervised learning as a similarity measurement to cluster the compressed streams into subgroups. The streams in every subgroup have been interleaved, followed by the first derivative to minimize the values and increase the redundancy. After that, two rotation steps have been applied to rearrange the readings in a more consecutive format before applying a developed dynamic run length. Finally, entropy coding is performed. Both mathematical and empirical experiments proved the significant improvement of the compressed streams entropy (i.e. almost reduced by half) and the resultant compression ratio (i.e. up to 50%).