Hybrid Attention for Robust RGB-T Pedestrian Detection in Real-World Conditions

TL;DR

This paper introduces a Hybrid Attention mechanism for RGB-T pedestrian detection that is robust to partial overlaps and sensor failures, improving performance in real-world autonomous driving scenarios.

Contribution

The paper proposes a novel Hybrid Attention module and an improved fusion algorithm that handle partial overlap and sensor failure, with a mobile-friendly backbone for embedded systems.

Findings

Outperforms state-of-the-art methods in simulated real-world scenarios.

Robust against partial overlap and sensor failure during inference.

Effective in resource-constrained embedded systems.

Abstract

Multispectral pedestrian detection has gained significant attention in recent years, particularly in autonomous driving applications. To address the challenges posed by adversarial illumination conditions, the combination of thermal and visible images has demonstrated its advantages. However, existing fusion methods rely on the critical assumption that the RGB-Thermal (RGB-T) image pairs are fully overlapping. These assumptions often do not hold in real-world applications, where only partial overlap between images can occur due to sensors configuration. Moreover, sensor failure can cause loss of information in one modality. In this paper, we propose a novel module called the Hybrid Attention (HA) mechanism as our main contribution to mitigate performance degradation caused by partial overlap and sensor failure, i.e. when at least part of the scene is acquired by only one sensor. We propose an improved RGB-T fusion algorithm, robust against partial overlap and sensor failure encountered during inference in real-world applications. We also leverage a mobile-friendly backbone to cope with resource constraints in embedded systems. We conducted experiments by simulating various partial overlap and sensor failure scenarios to evaluate the performance of our proposed method. The results demonstrate that our approach outperforms state-of-the-art methods, showcasing its superiority in handling real-world challenges.