DIM-SUM: Dynamic IMputation for Smart Utility Management

TL;DR

DIM-SUM is a preprocessing framework that improves time series imputation in real-world utility data by handling complex missing patterns, outperforming traditional methods in accuracy and efficiency.

Contribution

It introduces a novel approach combining pattern clustering and adaptive masking with theoretical guarantees to better handle real missing data patterns.

Findings

Outperforms traditional imputation methods in accuracy and speed.

Achieves similar accuracy with less training data and processing time.

Doubles the accuracy of a large pre-trained model with less inference time.

Abstract

Time series imputation models have traditionally been developed using complete datasets with artificial masking patterns to simulate missing values. However, in real-world infrastructure monitoring, practitioners often encounter datasets where large amounts of data are missing and follow complex, heterogeneous patterns. We introduce DIM-SUM, a preprocessing framework for training robust imputation models that bridges the gap between artificially masked training data and real missing patterns. DIM-SUM combines pattern clustering and adaptive masking strategies with theoretical learning guarantees to handle diverse missing patterns actually observed in the data. Through extensive experiments on over 2 billion readings from California water districts, electricity datasets, and benchmarks, we demonstrate that DIM-SUM outperforms traditional methods by reaching similar accuracy with lower processing time and significantly less training data. When compared against a large pre-trained model, DIM-SUM averages 2x higher accuracy with significantly less inference time.