SemLT3D: Semantic-Guided Expert Distillation for Camera-only Long-Tailed 3D Object Detection

TL;DR

SemLT3D introduces a semantic-guided expert distillation framework that enhances long-tail 3D object detection from camera data by leveraging semantic priors and CLIP-informed features.

Contribution

It proposes a novel semantic-guided expert routing and distillation approach to improve recognition of rare classes in camera-only 3D detection.

Findings

Improves detection accuracy for underrepresented classes.

Enhances robustness to appearance variations and challenging cases.

Leverages semantic priors and CLIP features for better feature discrimination.

Abstract

Camera-only 3D object detection has emerged as a cost-effective and scalable alternative to LiDAR for autonomous driving, yet existing methods primarily prioritize overall performance while overlooking the severe long-tail imbalance inherent in real-world datasets. In practice, many rare but safety-critical categories such as children, strollers, or emergency vehicles are heavily underrepresented, leading to biased learning and degraded performance. This challenge is further exacerbated by pronounced inter-class ambiguity (e.g., visually similar subclasses) and substantial intra-class diversity (e.g., objects varying widely in appearance, scale, pose, or context), which together hinder reliable long-tail recognition. In this work, we introduce SemLT3D, a Semantic-Guided Expert Distillation framework designed to enrich the representation space for underrepresented classes through semantic priors. SemLT3D consists of: (1) a language-guided mixture-of-experts module that routes 3D queries to specialized experts according to their semantic affinity, enabling the model to better disentangle confusing classes and specialize on tail distributions; and (2) a semantic projection distillation pipeline that aligns 3D queries with CLIP-informed 2D semantics, producing more coherent and discriminative features across diverse visual manifestations. Although motivated by long-tail imbalance, the semantically structured learning in SemLT3D also improves robustness under broader appearance variations and challenging corner cases, offering a principled step toward more reliable camera-only 3D perception.