Model Optimization for Multi-Camera 3D Detection and Tracking

TL;DR

This paper evaluates and optimizes Sparse4D, a multi-camera 3D detection and tracking framework, focusing on input frame rate reduction, quantization, and mixed-precision fine-tuning to improve speed and stability in indoor multi-camera perception.

Contribution

It introduces stability-aware validation for identity tracking and demonstrates effective speed-accuracy trade-offs through selective quantization and mixed-precision techniques.

Findings

Sparse4D remains stable at moderate FPS reductions.

Selective quantization improves speed with minimal accuracy loss.

Mixed-precision fine-tuning reduces latency but can affect identity stability.

Abstract

Outside-in multi-camera perception is increasingly important in indoor environments, where networks of static cameras must support multi-target tracking under occlusion and heterogeneous viewpoints. We evaluate Sparse4D, a query-based spatiotemporal 3D detection and tracking framework that fuses multi-view features in a shared world frame and propagates sparse object queries via instance memory. We study reduced input frame rates, post-training quantization (INT8 and FP8), transfer to the WILDTRACK benchmark, and Transformer Engine mixed-precision fine-tuning. To better capture identity stability, we report Average Track Duration (AvgTrackDur), which measures identity persistence in seconds. Sparse4D remains stable under moderate FPS reductions, but below 2 FPS, identity association collapses even when detections are stable. Selective quantization of the backbone and neck offers the best speed-accuracy trade-off, while attention-related modules are consistently sensitive to low precision. On WILDTRACK, low-FPS pretraining yields large zero-shot gains over the base checkpoint, while small-scale fine-tuning provides limited additional benefit. Transformer Engine mixed precision reduces latency and improves camera scalability, but can destabilize identity propagation, motivating stability-aware validation.