Strip-Fusion: Spatiotemporal Fusion for Multispectral Pedestrian Detection

TL;DR

Strip-Fusion introduces a spatiotemporal fusion network for multispectral pedestrian detection that is robust to misalignment, lighting variations, and occlusions, achieving improved performance on key benchmarks.

Contribution

The paper proposes a novel spatiotemporal fusion network with adaptive convolutions and a divergence loss to enhance multispectral pedestrian detection under challenging conditions.

Findings

Improves detection accuracy on KAIST and CVC-14 benchmarks.

Achieves better performance under occlusion and misalignment.

Reduces false positives with a new post-processing algorithm.

Abstract

Pedestrian detection is a critical task in robot perception. Multispectral modalities (visible light and thermal) can boost pedestrian detection performance by providing complementary visual information. Several gaps remain with multispectral pedestrian detection methods. First, existing approaches primarily focus on spatial fusion and often neglect temporal information. Second, RGB and thermal image pairs in multispectral benchmarks may not always be perfectly aligned. Pedestrians are also challenging to detect due to varying lighting conditions, occlusion, etc. This work proposes Strip-Fusion, a spatial-temporal fusion network that is robust to misalignment in input images, as well as varying lighting conditions and heavy occlusions. The Strip-Fusion pipeline integrates temporally adaptive convolutions to dynamically weigh spatial-temporal features, enabling our model to better capture pedestrian motion and context over time. A novel Kullback-Leibler divergence loss was designed to mitigate modality imbalance between visible and thermal inputs, guiding feature alignment toward the more informative modality during training. Furthermore, a novel post-processing algorithm was developed to reduce false positives. Extensive experimental results show that our method performs competitively for both the KAIST and the CVC-14 benchmarks. We also observed significant improvements compared to previous state-of-the-art on challenging conditions such as heavy occlusion and misalignment.