Distilling Future Temporal Knowledge with Masked Feature Reconstruction for 3D Object Detection

TL;DR

This paper introduces FTKD, a novel knowledge distillation method that transfers future temporal information from offline to online models in 3D object detection, improving accuracy without extra inference cost.

Contribution

It proposes a future-aware feature reconstruction and logit distillation approach to effectively transfer future knowledge in temporal 3D detection models.

Findings

Achieves up to 1.3 mAP and 1.3 NDS improvements on nuScenes

Provides the most accurate velocity estimation

Does not increase inference cost

Abstract

Camera-based temporal 3D object detection has shown impressive results in autonomous driving, with offline models improving accuracy by using future frames. Knowledge distillation (KD) can be an appealing framework for transferring rich information from offline models to online models. However, existing KD methods overlook future frames, as they mainly focus on spatial feature distillation under strict frame alignment or on temporal relational distillation, thereby making it challenging for online models to effectively learn future knowledge. To this end, we propose a sparse query-based approach, Future Temporal Knowledge Distillation (FTKD), which effectively transfers future frame knowledge from an offline teacher model to an online student model. Specifically, we present a future-aware feature reconstruction strategy to encourage the student model to capture future features without strict frame alignment. In addition, we further introduce future-guided logit distillation to leverage the teacher's stable foreground and background context. FTKD is applied to two high-performing 3D object detection baselines, achieving up to 1.3 mAP and 1.3 NDS gains on the nuScenes dataset, as well as the most accurate velocity estimation, without increasing inference cost.