Quantifying Quality of Class-Conditional Generative Models in Time-Series Domain

TL;DR

This paper introduces new metrics, InceptionTime Score and Frechet InceptionTime Distance, to evaluate the quality of class-conditional generative models in time-series data, addressing a key gap in model assessment.

Contribution

It proposes novel evaluation metrics for time-series generative models and validates their effectiveness through extensive experiments on diverse datasets.

Findings

ITS and FITD accurately assess model quality

Proposed metrics outperform existing evaluation methods

Combination of ITS, FITD, and TSTR provides reliable assessment

Abstract

Generative models are designed to address the data scarcity problem. Even with the exploding amount of data, due to computational advancements, some applications (e.g., health care, weather forecast, fault detection) still suffer from data insufficiency, especially in the time-series domain. Thus generative models are essential and powerful tools, but they still lack a consensual approach for quality assessment. Such deficiency hinders the confident application of modern implicit generative models on time-series data. Inspired by assessment methods on the image domain, we introduce the InceptionTime Score (ITS) and the Frechet InceptionTime Distance (FITD) to gauge the qualitative performance of class conditional generative models on the time-series domain. We conduct extensive experiments on 80 different datasets to study the discriminative capabilities of proposed metrics alongside two existing evaluation metrics: Train on Synthetic Test on Real (TSTR) and Train on Real Test on Synthetic (TRTS). Extensive evaluation reveals that the proposed assessment method, i.e., ITS and FITD in combination with TSTR, can accurately assess class-conditional generative model performance.