MASAR: Motion-Appearance Synergy Refinement for Joint Detection and Trajectory Forecasting

TL;DR

MASAR is a novel framework that enhances joint 3D detection and trajectory forecasting in autonomous driving by effectively integrating appearance and motion cues through a differentiable, object-centric spatio-temporal mechanism, improving long-term predictions.

Contribution

It introduces MASAR, a fully differentiable, appearance-motion synergy framework compatible with transformer-based detectors, capturing long-term dependencies for better trajectory forecasting.

Findings

Over 20% improvement in minADE and minFDE on nuScenes

Effective integration of appearance and motion cues

Maintains robust detection performance

Abstract

Classical autonomous driving systems connect perception and prediction modules via hand-crafted bounding-box interfaces, limiting information flow and propagating errors to downstream tasks. Recent research aims to develop end-to-end models that jointly address perception and prediction; however, they often fail to fully exploit the synergy between appearance and motion cues, relying mainly on short-term visual features. We follow the idea of "looking backward to look forward", and propose MASAR, a novel fully differentiable framework for joint 3D detection and trajectory forecasting compatible with any transformer-based 3D detector. MASAR employs an object-centric spatio-temporal mechanism that jointly encodes appearance and motion features. By predicting past trajectories and refining them using guidance from appearance cues, MASAR captures long-term temporal dependencies that enhance future trajectory forecasting. Experiments conducted on the nuScenes dataset demonstrate MASAR's effectiveness, showing improvements of over 20% in minADE and minFDE while maintaining robust detection performance. Code and models are available at https://github.com/aminmed/MASAR.