Extract Dynamic Information To Improve Time Series Modeling: a Case Study with Scientific Workflow

TL;DR

This paper presents techniques for extracting dynamic information from scientific workflows to enhance time series prediction accuracy, demonstrating significant error reduction in data transfer time modeling.

Contribution

The work introduces methods to incorporate dynamic workflow information into time series models, improving prediction accuracy over static feature-based approaches.

Findings

Reduced prediction error by 12% using recent event matching.

Achieved 44% error reduction with application-specific data production features.

Demonstrated generalizability of techniques to other applications.

Abstract

In modeling time series data, we often need to augment the existing data records to increase the modeling accuracy. In this work, we describe a number of techniques to extract dynamic information about the current state of a large scientific workflow, which could be generalized to other types of applications. The specific task to be modeled is the time needed for transferring a file from an experimental facility to a data center. The key idea of our approach is to find recent past data transfer events that match the current event in some ways. Tests showed that we could identify recent events matching some recorded properties and reduce the prediction error by about 12% compared to the similar models with only static features. We additionally explored an application specific technique to extract information about the data production process, and was able to reduce the average prediction error by 44%.