Towards a Deeper Understanding of Transformer for Residential Non-intrusive Load Monitoring

TL;DR

This paper investigates how various hyper-parameters affect transformer models in residential Non-Intrusive Load Monitoring, providing insights and guidelines to optimize their performance.

Contribution

It offers a comprehensive analysis of hyper-parameter impacts on transformer performance in NILM and proposes optimized settings that improve accuracy over existing models.

Findings

Optimal hyper-parameters identified for transformer models in NILM

Transformer with tuned hyper-parameters outperforms existing models

Guidelines for hyper-parameter selection in NILM transformers

Abstract

Transformer models have demonstrated impressive performance in Non-Intrusive Load Monitoring (NILM) applications in recent years. Despite their success, existing studies have not thoroughly examined the impact of various hyper-parameters on model performance, which is crucial for advancing high-performing transformer models. In this work, a comprehensive series of experiments have been conducted to analyze the influence of these hyper-parameters in the context of residential NILM. This study delves into the effects of the number of hidden dimensions in the attention layer, the number of attention layers, the number of attention heads, and the dropout ratio on transformer performance. Furthermore, the role of the masking ratio has explored in BERT-style transformer training, providing a detailed investigation into its impact on NILM tasks. Based on these experiments, the optimal hyper-parameters have been selected and used them to train a transformer model, which surpasses the performance of existing models. The experimental findings offer valuable insights and guidelines for optimizing transformer architectures, aiming to enhance their effectiveness and efficiency in NILM applications. It is expected that this work will serve as a foundation for future research and development of more robust and capable transformer models for NILM.