Creating a Causally Grounded Rating Method for Assessing the Robustness of AI Models for Time-Series Forecasting

TL;DR

This paper introduces a causally grounded rating framework to evaluate the robustness of AI time-series forecasting models against input perturbations, aiding stakeholders in model comparison without needing access to internal details.

Contribution

We develop a novel causally grounded evaluation framework for assessing robustness of time-series models under various noisy scenarios, applicable in black-box settings.

Findings

Multi-modal and time-series-specific FMs are more robust than general-purpose models.

The rating framework effectively differentiates model robustness in noisy input conditions.

User study shows ratings help users compare model robustness more easily.

Abstract

AI models, including both time-series-specific and general-purpose Foundation Models (FMs), have demonstrated strong potential in time-series forecasting across sectors like finance. However, these models are highly sensitive to input perturbations, which can lead to prediction errors and undermine trust among stakeholders, including investors and analysts. To address this challenge, we propose a causally grounded rating framework to systematically evaluate model robustness by analyzing statistical and confounding biases under various noisy and erroneous input scenarios. Our framework is applied to a large-scale experimental setup involving stock price data from multiple industries and evaluates both uni-modal and multi-modal models, including Vision Transformer-based (ViT) models and FMs. We introduce six types of input perturbations and twelve data distributions to assess model performance. Results indicate that multi-modal and time-series-specific FMs demonstrate greater robustness and accuracy compared to general-purpose models. Further, to validate our framework's usability, we conduct a user study showcasing time-series models' prediction errors along with our computed ratings. The study confirms that our ratings reduce the difficulty for users in comparing the robustness of different models. Our findings can help stakeholders understand model behaviors in terms of robustness and accuracy for better decision-making even without access to the model weights and training data, i.e., black-box settings.