Scaled and Inter-token Relation Enhanced Transformer for Sample-restricted Residential NILM

TL;DR

This paper introduces a novel transformer architecture with inter-token relation enhancement and dynamic temperature tuning, improving performance on small datasets for Non-Intrusive Load Monitoring by better capturing inter-token dependencies.

Contribution

The paper proposes two innovations—relation enhancement and adaptive temperature tuning—to improve transformer performance on small datasets for NILM tasks.

Findings

Outperforms baseline transformers by 10-15% in F1 score

Effective on small datasets like REDD

Enhances inter-token relationship modeling

Abstract

Transformers have demonstrated exceptional performance across various domains due to their self-attention mechanism, which captures complex relationships in data. However, training on smaller datasets poses challenges, as standard attention mechanisms can over-smooth attention scores and overly prioritize intra-token relationships, reducing the capture of meaningful inter-token dependencies critical for tasks like Non-Intrusive Load Monitoring (NILM). To address this, we propose a novel transformer architecture with two key innovations: inter-token relation enhancement and dynamic temperature tuning. The inter-token relation enhancement mechanism removes diagonal entries in the similarity matrix to improve attention focus on inter-token relations. The dynamic temperature tuning mechanism, a learnable parameter, adapts attention sharpness during training, preventing over-smoothing and enhancing sensitivity to token relationships. We validate our method on the REDD dataset and show that it outperforms the original transformer and state-of-the-art models by 10-15\% in F1 score across various appliance types, demonstrating its efficacy for training on smaller datasets.