Finding Islands of Predictability in Action Forecasting

TL;DR

This paper introduces a novel approach for dense action forecasting that dynamically selects abstraction levels during prediction, identifying 'islands of predictability' to improve accuracy in long-term action sequence prediction.

Contribution

It proposes a combined Bayesian neural network and hierarchical segmentation model that adaptively chooses abstraction levels, enhancing long-term action forecasting accuracy.

Findings

Significant accuracy improvements over state-of-the-art methods.

Effective identification of high-confidence 'islands' in future predictions.

Maintains detailed predictions in small segments while abstracting elsewhere.

Abstract

We address dense action forecasting: the problem of predicting future action sequence over long durations based on partial observation. Our key insight is that future action sequences are more accurately modeled with variable, rather than one, levels of abstraction, and that the optimal level of abstraction can be dynamically selected during the prediction process. Our experiments show that most parts of future action sequences can be predicted confidently in fine detail only in small segments of future frames, which are effectively ``islands'' of high model prediction confidence in a ``sea'' of uncertainty. We propose a combination Bayesian neural network and hierarchical convolutional segmentation model to both accurately predict future actions and optimally select abstraction levels. We evaluate this approach on standard datasets against existing state-of-the-art systems and demonstrate that our ``islands of predictability'' approach maintains fine-grained action predictions while also making accurate abstract predictions where systems were previously unable to do so, and thus results in substantial, monotonic increases in accuracy.