DINO-CoDT: Multi-class Collaborative Detection and Tracking with Vision Foundation Models

TL;DR

This paper introduces DINO-CoDT, a multi-class collaborative detection and tracking framework utilizing vision foundation models, with novel modules for feature learning, re-identification, and adaptive track management, significantly improving accuracy in diverse road scenarios.

Contribution

The paper presents a new multi-class collaborative detection and tracking framework with innovative modules for feature fusion, re-identification, and adaptive tracking, addressing limitations of prior vehicle-focused methods.

Findings

Outperforms state-of-the-art in detection accuracy

Reduces ID switch errors effectively

Enhances tracking robustness across diverse object classes

Abstract

Collaborative perception plays a crucial role in enhancing environmental understanding by expanding the perceptual range and improving robustness against sensor failures, which primarily involves collaborative 3D detection and tracking tasks. The former focuses on object recognition in individual frames, while the latter captures continuous instance tracklets over time. However, existing works in both areas predominantly focus on the vehicle superclass, lacking effective solutions for both multi-class collaborative detection and tracking. This limitation hinders their applicability in real-world scenarios, which involve diverse object classes with varying appearances and motion patterns. To overcome these limitations, we propose a multi-class collaborative detection and tracking framework tailored for diverse road users. We first present a detector with a global spatial attention fusion (GSAF) module, enhancing multi-scale feature learning for objects of varying sizes. Next, we introduce a tracklet RE-IDentification (REID) module that leverages visual semantics with a vision foundation model to effectively reduce ID SWitch (IDSW) errors, in cases of erroneous mismatches involving small objects like pedestrians. We further design a velocity-based adaptive tracklet management (VATM) module that adjusts the tracking interval dynamically based on object motion. Extensive experiments on the V2X-Real and OPV2V datasets show that our approach significantly outperforms existing state-of-the-art methods in both detection and tracking accuracy.