smoothEM: a new approach for the simultaneous assessment of smooth patterns and spikes

TL;DR

smoothEM is a novel method combining spline smoothing and EM algorithm to simultaneously identify spikes and estimate smooth patterns in functional data, demonstrated on heatwave and electricity consumption datasets.

Contribution

It introduces a new approach that integrates regularized spline smoothing with EM to detect spikes and estimate smooth trends, with proven consistency and robust performance.

Findings

Successfully identifies spikes and smooth trends in real datasets.

Proves consistency of EM estimates under certain error assumptions.

Demonstrates robustness to violations of assumptions through simulations.

Abstract

We consider functional data where an underlying smooth curve is composed not just with errors, but also with irregular spikes. We propose an approach that, combining regularized spline smoothing and an Expectation-Maximization algorithm, allows one to both identify spikes and estimate the smooth component. Imposing some assumptions on the error distribution, we prove consistency of EM estimates. Next, we demonstrate the performance of our proposal on finite samples and its robustness to assumptions violations through simulations. Finally, we apply our proposal to data on the annual heatwaves index in the US and on weekly electricity consumption in Ireland. In both datasets, we are able to characterize underlying smooth trends and to pinpoint irregular/extreme behaviors.