Investigating the Performance Gap between Testing on Real and Denoised Aggregates in Non-Intrusive Load Monitoring

TL;DR

This study investigates the performance differences of NILM algorithms when tested on real-world versus denoised aggregate signals, revealing significant performance gaps influenced by signal complexity and noise levels.

Contribution

The paper provides the first comprehensive analysis of the performance gap between real and denoised aggregate testing in NILM, highlighting the impact of noise and complexity.

Findings

Algorithms perform significantly better on denoised signals.

Performance gap increases with higher noise levels.

All appliance types are affected by this phenomenon.

Abstract

Prudent and meaningful performance evaluation of algorithms is essential for the progression of any research field. In the field of Non-Intrusive Load Monitoring (NILM), performance evaluation can be conducted on real-world aggregate signals, provided by smart energy meters or artificial superpositions of individual load signals (i.e., denoised aggregates). It has long been suspected that testing on these denoised aggregates provides better evaluation results mainly due to the the fact that the signal is less complex. Complexity in real-world aggregate signals increases with the number of unknown/untracked load. Although this is a known performance reporting problem, an investigation in the actual performance gap between real and denoised testing is still pending. In this paper, we examine the performance gap between testing on real-world and denoised aggregates with the aim of bringing clarity into this matter. Starting with an assessment of noise levels in datasets, we find significant differences in test cases. We give broad insights into our evaluation setup comprising three load disaggregation algorithms, two of them relying on neural network architectures. The results presented in this paper, based on studies covering three scenarios with ascending noise levels, show a strong tendency towards load disaggregation algorithms providing significantly better performance on denoised aggregate signals. A closer look into the outcome of our studies reveals that all appliance types could be subject to this phenomenon. We conclude the paper by discussing aspects that could be causing these considerable gaps between real and denoised testing in NILM.