Why Did This Model Forecast This Future? Closed-Form Temporal Saliency Towards Causal Explanations of Probabilistic Forecasts

TL;DR

This paper introduces a closed-form method for identifying salient observed windows in probabilistic forecasting models, enabling better explanations of model predictions without needing internal model access.

Contribution

It extends information-theoretic saliency to forecasting by deriving a closed-form solution based on differential entropy, applicable to common density functions.

Findings

Successfully recovers salient observed windows in head pose forecasting

Provides a model-agnostic, closed-form saliency map method

Demonstrates effectiveness on synthesized conversation data

Abstract

Forecasting tasks surrounding the dynamics of low-level human behavior are of significance to multiple research domains. In such settings, methods for explaining specific forecasts can enable domain experts to gain insights into the predictive relationships between behaviors. In this work, we introduce and address the following question: given a probabilistic forecasting model how can we identify observed windows that the model considers salient when making its forecasts? We build upon a general definition of information-theoretic saliency grounded in human perception and extend it to forecasting settings by leveraging a crucial attribute of the domain: a single observation can result in multiple valid futures. We propose to express the saliency of an observed window in terms of the differential entropy of the resulting predicted future distribution. In contrast to existing methods that either require explicit training of the saliency mechanism or access to the internal states of the forecasting model, we obtain a closed-form solution for the saliency map for commonly used density functions in probabilistic forecasting. We empirically demonstrate how our framework can recover salient observed windows from head pose features for the sample task of speaking-turn forecasting using a synthesized conversation dataset.