Validation of machine learning based scenario generators

TL;DR

This paper presents new validation techniques for machine learning-based scenario generators, focusing on dependency checks and memorization detection, validated through real market data experiments.

Contribution

It introduces a novel memorization ratio and applies existing dependency tests to validate ML scenario generators, addressing unique challenges in data-driven models.

Findings

Dependency checks reveal data-driven dependencies in ML models.

Memorization ratio effectively detects overfitting in scenario generation.

Real market data experiments validate the proposed methods.

Abstract

Machine learning (ML) methods are becoming increasingly important in the design economic scenario generators for internal models. Validation of data-driven models differs from classical theory-based models. We discuss two novel aspects of such a validation: first, checking dependencies between risk factors and second, detecting unwanted memorization effects. The first task becomes necessary since in ML-based methods dependencies are no longer derived from a financial-mathematical theory but are driven by data. The need for the latter task arises since it cannot be ruled out that ML-based models merely reproduce the empirical data rather than generating new scenarios. To address the first issue, we propose to use an existing test from the literature. For the second issue, we introduce and discuss a novel memorization ratio. Numerical experiments based on real market data are included and an autoencoder-based scenario generator is validated with these two methods.