How do Mixture Density RNNs Predict the Future?

TL;DR

This paper analyzes how mixture density RNNs predict multiple future possibilities, revealing their role in modeling stochastic events and scenario-based rules, which enhances understanding of their internal representations.

Contribution

The study provides insights into the internal decomposition of mixture density RNNs, showing how their Gaussian components model different stochastic events and scenarios.

Findings

Gaussian components model different stochastic events

Components separate scenarios with different rules

Enhances understanding of MD-RNN internal representations

Abstract

Gaining a better understanding of how and what machine learning systems learn is important to increase confidence in their decisions and catalyze further research. In this paper, we analyze the predictions made by a specific type of recurrent neural network, mixture density RNNs (MD-RNNs). These networks learn to model predictions as a combination of multiple Gaussian distributions, making them particularly interesting for problems where a sequence of inputs may lead to several distinct future possibilities. An example is learning internal models of an environment, where different events may or may not occur, but where the average over different events is not meaningful. By analyzing the predictions made by trained MD-RNNs, we find that their different Gaussian components have two complementary roles: 1) Separately modeling different stochastic events and 2) Separately modeling scenarios governed by different rules. These findings increase our understanding of what is learned by predictive MD-RNNs, and open up new research directions for further understanding how we can benefit from their self-organizing model decomposition.