Divide and Merge: Motion and Semantic Learning in End-to-End Autonomous Driving

TL;DR

This paper introduces Neural-Bayes motion decoding, a novel approach that separates semantic and motion learning in end-to-end autonomous driving, improving perception, prediction, and planning performance by addressing negative transfer issues.

Contribution

It proposes a divide and merge framework with parallel motion and semantic learning, enhancing multi-task learning in autonomous driving.

Findings

Improved perception, prediction, and planning performance on nuScenes dataset.

Effective separation of semantic and motion learning reduces negative transfer.

Code implementation available online.

Abstract

Perceiving the environment and its changes over time corresponds to two fundamental yet heterogeneous types of information: semantics and motion. Previous end-to-end autonomous driving works represent both types of information in a single feature vector. However, including motion related tasks, such as prediction and planning, impairs detection and tracking performance, a phenomenon known as negative transfer in multi-task learning. To address this issue, we propose Neural-Bayes motion decoding, a novel parallel detection, tracking, and prediction method that separates semantic and motion learning. Specifically, we employ a set of learned motion queries that operate in parallel with detection and tracking queries, sharing a unified set of recursively updated reference points. Moreover, we employ interactive semantic decoding to enhance information exchange in semantic tasks, promoting positive transfer. Experiments on the nuScenes dataset with UniAD and SparseDrive confirm the effectiveness of our divide and merge approach, resulting in performance improvements across perception, prediction, and planning. Our code is available at https://github.com/shenyinzhe/DMAD.