Recall to Predict: Grounding Motion Forecasting in Interpretable Motion Bank

TL;DR

This paper introduces a novel motion forecasting framework grounded in an interpretable motion bank, combining contrastive learning, a retrieval mechanism, and a structured decoding process to improve multi-modal accuracy.

Contribution

It proposes a new end-to-end differentiable architecture that uses a structured motion bank and explicit motion priors for interpretable and accurate motion prediction.

Findings

Achieves competitive multi-modal accuracy on Argoverse 2 and Waymo datasets.

Eliminates the black box issue of latent queries in motion forecasting.

Introduces a novel Anchor Retrieval Layer with a Dual-Level Gated Cross-Attention mechanism.

Abstract

Motion forecasting often requires trading interpretability for predictive accuracy. Standard anchor-based architectures rely on opaque latent queries that are highly prone to latent collapse, or naive trajectory sampling that limits multi-modal diversity. We propose an end-to-end differentiable framework that grounds predictions in a comprehensive "motion bank", a structured embedding space of physically realizable trajectories constructed via contrastive learning. Rather than regressing paths from a blank slate, our architecture dynamically retrieves explicit motion priors using a novel Anchor Retrieval Layer. This module adapts orthogonally initialized queries via a Dual-Level Gated Cross-Attention mechanism and executes discrete trajectory selection using a Straight-Through Gumbel-Softmax estimator to preserve continuous gradient flow. The retrieved semantically grounded anchors are then geometrically refined by a DETR-style decoder, optimized jointly with a Winner-Takes-All (WTA) kinematic Gaussian Mixture Model (GMM), a latent diversity penalty, and a soft-min weighted endpoint loss. By strictly conditioning the decoding phase on diverse, interpretable motion primitives, our approach eliminates the "black box" of standard latent queries while achieving competitive multi-modal accuracy on the Argoverse 2 and Waymo Open Motion datasets. Code is available at: https://github.com/abviv/recall2predict