Development of a Neural Network-based Method for Improved Imputation of Missing Values in Time Series Data by Repurposing DataWig

TL;DR

This paper introduces tsDataWig, a neural network-based method adapted from DataWig, designed to improve imputation of missing values in complex, heterogeneous time series data without strong assumptions about missing data mechanisms.

Contribution

The study presents tsDataWig, a novel adaptation of DataWig, specifically tailored for time series data, enhancing imputation accuracy and scalability over existing methods.

Findings

tsDataWig outperforms original DataWig in time series imputation tasks.

tsDataWig handles large, high-dimensional, heterogeneous datasets effectively.

The method does not rely on strong assumptions about missing data mechanisms.

Abstract

Time series data are observations collected over time intervals. Successful analysis of time series data captures patterns such as trends, cyclicity and irregularity, which are crucial for decision making in research, business, and governance. However, missing values in time series data occur often and present obstacles to successful analysis, thus they need to be filled with alternative values, a process called imputation. Although various approaches have been attempted for robust imputation of time series data, even the most advanced methods still face challenges including limited scalability, poor capacity to handle heterogeneous data types and inflexibility due to requiring strong assumptions of data missing mechanisms. Moreover, the imputation accuracy of these methods still has room for improvement. In this study, I developed tsDataWig (time-series DataWig) by modifying DataWig, a neural network-based method that possesses the capacity to process large datasets and heterogeneous data types but was designed for non-time series data imputation. Unlike the original DataWig, tsDataWig can directly handle values of time variables and impute missing values in complex time series datasets. Using one simulated and three different complex real-world time series datasets, I demonstrated that tsDataWig outperforms the original DataWig and the current state-of-the-art methods for time series data imputation and potentially has broad application due to not requiring strong assumptions of data missing mechanisms. This study provides a valuable solution for robustly imputing missing values in challenging time series datasets, which often contain millions of samples, high dimensional variables, and heterogeneous data types.