Trajectory Forecasting through Low-Rank Adaptation of Discrete Latent Codes

TL;DR

This paper introduces a novel trajectory forecasting method combining VQ-VAEs with low-rank adaptation of discrete latent codes, achieving state-of-the-art results by balancing diversity and fidelity in predictions.

Contribution

It proposes a flexible, instance-specific discrete latent space with low-rank updates for trajectory forecasting, enhancing reconstruction and diversity over prior models.

Findings

Achieves state-of-the-art performance on three benchmarks.

Improves trajectory diversity and accuracy.

Utilizes low-rank updates for flexible latent adaptation.

Abstract

Trajectory forecasting is crucial for video surveillance analytics, as it enables the anticipation of future movements for a set of agents, e.g. basketball players engaged in intricate interactions with long-term intentions. Deep generative models offer a natural learning approach for trajectory forecasting, yet they encounter difficulties in achieving an optimal balance between sampling fidelity and diversity. We address this challenge by leveraging Vector Quantized Variational Autoencoders (VQ-VAEs), which utilize a discrete latent space to tackle the issue of posterior collapse. Specifically, we introduce an instance-based codebook that allows tailored latent representations for each example. In a nutshell, the rows of the codebook are dynamically adjusted to reflect contextual information (i.e., past motion patterns extracted from the observed trajectories). In this way, the discretization process gains flexibility, leading to improved reconstructions. Notably, instance-level dynamics are injected into the codebook through low-rank updates, which restrict the customization of the codebook to a lower dimension space. The resulting discrete space serves as the basis of the subsequent step, which regards the training of a diffusion-based predictive model. We show that such a two-fold framework, augmented with instance-level discretization, leads to accurate and diverse forecasts, yielding state-of-the-art performance on three established benchmarks.