Utilizing Data Fingerprints for Privacy-Preserving Algorithm Selection in Time Series Classification: Performance and Uncertainty Estimation on Unseen Datasets

TL;DR

This paper introduces a privacy-preserving data fingerprint for time series datasets that enables effective algorithm performance prediction and uncertainty estimation without accessing raw data, streamlining algorithm selection.

Contribution

The work presents a novel data fingerprint method that estimates algorithm performance and uncertainty without training on the actual datasets, reducing computational costs and privacy concerns.

Findings

Accurately predicts algorithm performance on unseen datasets

Improves performance estimation accuracy by 7.32% on average

Provides reliable uncertainty estimates for algorithm selection

Abstract

The selection of algorithms is a crucial step in designing AI services for real-world time series classification use cases. Traditional methods such as neural architecture search, automated machine learning, combined algorithm selection, and hyperparameter optimizations are effective but require considerable computational resources and necessitate access to all data points to run their optimizations. In this work, we introduce a novel data fingerprint that describes any time series classification dataset in a privacy-preserving manner and provides insight into the algorithm selection problem without requiring training on the (unseen) dataset. By decomposing the multi-target regression problem, only our data fingerprints are used to estimate algorithm performance and uncertainty in a scalable and adaptable manner. Our approach is evaluated on the 112 University of California riverside benchmark datasets, demonstrating its effectiveness in predicting the performance of 35 state-of-the-art algorithms and providing valuable insights for effective algorithm selection in time series classification service systems, improving a naive baseline by 7.32% on average in estimating the mean performance and 15.81% in estimating the uncertainty.