Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection

TL;DR

This paper introduces an illumination-aware fusion method for multispectral pedestrian detection, improving robustness under varying lighting by adaptively merging color and thermal data based on illumination conditions.

Contribution

It proposes an illumination-aware network and adaptive fusion strategy within Faster R-CNN, achieving state-of-the-art performance on the KAIST benchmark.

Findings

Improved detection accuracy under challenging illumination conditions.

Effective fusion architecture comparable to state-of-the-art.

Validation on KAIST dataset confirms robustness and effectiveness.

Abstract

Multispectral images of color-thermal pairs have shown more effective than a single color channel for pedestrian detection, especially under challenging illumination conditions. However, there is still a lack of studies on how to fuse the two modalities effectively. In this paper, we deeply compare six different convolutional network fusion architectures and analyse their adaptations, enabling a vanilla architecture to obtain detection performances comparable to the state-of-the-art results. Further, we discover that pedestrian detection confidences from color or thermal images are correlated with illumination conditions. With this in mind, we propose an Illumination-aware Faster R-CNN (IAF RCNN). Specifically, an Illumination-aware Network is introduced to give an illumination measure of the input image. Then we adaptively merge color and thermal sub-networks via a gate function defined over the illumination value. The experimental results on KAIST Multispectral Pedestrian Benchmark validate the effectiveness of the proposed IAF R-CNN.