RMT-PPAD: Real-time Multi-task Learning for Panoptic Perception in Autonomous Driving

TL;DR

This paper introduces RMT-PPAD, a real-time transformer-based multi-task model for panoptic perception in autonomous driving, achieving state-of-the-art accuracy and efficiency on the BDD100K dataset.

Contribution

The work presents a lightweight, adaptive multi-task learning framework with novel modules for feature fusion and segmentation, addressing negative transfer and label inconsistency issues.

Findings

Achieves 84.9% mAP50 in object detection

Attains 92.6% mIoU in drivable area segmentation

Runs at 32.6 FPS in real-time

Abstract

Autonomous driving systems rely on panoptic driving perception that requires both precision and real-time performance. In this work, we propose RMT-PPAD, a real-time, transformer-based multi-task model that jointly performs object detection, drivable area segmentation, and lane line segmentation. We introduce a lightweight module, a gate control with an adapter to adaptively fuse shared and task-specific features, effectively alleviating negative transfer between tasks. Additionally, we design an adaptive segmentation decoder to learn the weights over multi-scale features automatically during the training stage. This avoids the manual design of task-specific structures for different segmentation tasks. We also identify and resolve the inconsistency between training and testing labels in lane line segmentation. This allows fairer evaluation. Experiments on the BDD100K dataset demonstrate that RMT-PPAD achieves state-of-the-art results with mAP50 of 84.9% and Recall of 95.4% for object detection, mIoU of 92.6% for drivable area segmentation, and IoU of 56.8% and accuracy of 84.7% for lane line segmentation. The inference speed reaches 32.6 FPS. Moreover, we introduce real-world scenarios to evaluate RMT-PPAD performance in practice. The results show that RMT-PPAD consistently delivers stable performance. The source codes and pre-trained models are released at https://github.com/JiayuanWang-JW/RMT-PPAD.