Cross-Modality Proposal-guided Feature Mining for Unregistered RGB-Thermal Pedestrian Detection

TL;DR

This paper introduces a novel method for RGB-T pedestrian detection that effectively handles unregistered, misaligned image pairs by predicting separate pedestrian locations and mining cross-modality features, improving robustness in real-world scenarios.

Contribution

The paper proposes a cross-modality proposal-guided feature mining mechanism and a two-stream dense detector specifically designed for unregistered RGB-T image pairs, addressing alignment issues.

Findings

Effective detection of unregistered pedestrians with different shifts.

Robustness to misalignment demonstrated through experimental results.

Improved performance over existing methods in unaligned scenarios.

Abstract

RGB-Thermal (RGB-T) pedestrian detection aims to locate the pedestrians in RGB-T image pairs to exploit the complementation between the two modalities for improving detection robustness in extreme conditions. Most existing algorithms assume that the RGB-T image pairs are well registered, while in the real world they are not aligned ideally due to parallax or different field-of-view of the cameras. The pedestrians in misaligned image pairs may locate at different positions in two images, which results in two challenges: 1) how to achieve inter-modality complementation using spatially misaligned RGB-T pedestrian patches, and 2) how to recognize the unpaired pedestrians at the boundary. To deal with these issues, we propose a new paradigm for unregistered RGB-T pedestrian detection, which predicts two separate pedestrian locations in the RGB and thermal images, respectively. Specifically, we propose a cross-modality proposal-guided feature mining (CPFM) mechanism to extract the two precise fusion features for representing the pedestrian in the two modalities, even if the RGB-T image pair is unaligned. It enables us to effectively exploit the complementation between the two modalities. With the CPFM mechanism, we build a two-stream dense detector; it predicts the two pedestrian locations in the two modalities based on the corresponding fusion feature mined by the CPFM mechanism. Besides, we design a data augmentation method, named Homography, to simulate the discrepancy in scales and views between images. We also investigate two non-maximum suppression (NMS) methods for post-processing. Favorable experimental results demonstrate the effectiveness and robustness of our method in dealing with unregistered pedestrians with different shifts.