Signature-Kernel Based Evaluation Metrics for Robust Probabilistic and Tail-Event Forecasting

TL;DR

This paper introduces two kernel-based metrics, Sig-MMD and CSig-MMD, to evaluate probabilistic forecasts more reliably, especially for tail events, by capturing complex dependencies and handling missing data.

Contribution

It proposes novel signature kernel-based metrics that improve tail-event sensitivity and dependency capturing in probabilistic forecast evaluation.

Findings

Sig-MMD captures complex dependencies in forecasts.

CSig-MMD emphasizes tail-event prediction accuracy.

Metrics are robust to missing data.

Abstract

Probabilistic forecasting is increasingly critical across high-stakes domains, from finance and epidemiology to climate science. However, current evaluation frameworks lack a consensus metric and suffer from two critical flaws: they often assume independence across time steps or variables, and they demonstrably lack sensitivity to tail events, the very occurrences that are most pivotal in real-world decision-making. To address these limitations, we propose two kernel-based metrics: the signature maximum mean discrepancy (Sig-MMD) and our novel censored Sig-MMD (CSig-MMD). By leveraging the signature kernel, these metrics capture complex inter-variate and inter-temporal dependencies and remain robust to missing data. Furthermore, CSig-MMD introduces a censoring scheme that prioritizes a forecaster's capability to predict tail events while strictly maintaining properness, a vital property for a good scoring rule. These metrics enable a more reliable evaluation of direct multi-step forecasting, facilitating the development of more robust probabilistic algorithms.