Learning from limited temporal data: Dynamically sparse historical functional linear models with applications to Earth science

TL;DR

This paper introduces a flexible, dynamic sparse functional linear model that estimates time-varying lags in relationships between variables, demonstrated through hydrological data analysis.

Contribution

It simplifies existing models by using a rectangular coefficient structure and hierarchical estimation to allow lag variability over time.

Findings

Accurately estimates dynamic time lags in simulated data.

Effectively extracts rainfall-runoff processes from real hydrological data.

Demonstrates improved model flexibility over previous methods.

Abstract

Scientists and statisticians often want to learn about the complex relationships that connect two time-varying variables. Recent work on sparse functional historical linear models confirms that they are promising for this purpose, but several notable limitations exist. Most importantly, previous works have imposed sparsity on the historical coefficient function, but have not allowed the sparsity, hence lag, to vary with time. We simplify the framework of sparse functional historical linear models by using a rectangular coefficient structure along with Whittaker smoothing, then reduce the assumptions of the previous frameworks by estimating the dynamic time lag from a hierarchical coefficient structure. We motivate our study by aiming to extract the physical rainfall-runoff processes hidden within hydrological data. We show the promise and accuracy of our method using eight simulation studies, further justified by two real sets of hydrological data.