Multispectral Pedestrian Detection via Simultaneous Detection and Segmentation

TL;DR

This paper introduces a multispectral detection network that combines detection and segmentation to improve pedestrian detection in low-light conditions, significantly outperforming existing methods on the KAIST dataset.

Contribution

The paper proposes a novel fusion architecture that jointly optimizes detection and segmentation, and provides a sanitized dataset annotation to reduce errors impacting model performance.

Findings

Outperforms state-of-the-art on KAIST dataset

Joint detection and segmentation improves accuracy

Sanitized annotations help reduce training errors

Abstract

Multispectral pedestrian detection has attracted increasing attention from the research community due to its crucial competence for many around-the-clock applications (e.g., video surveillance and autonomous driving), especially under insufficient illumination conditions. We create a human baseline over the KAIST dataset and reveal that there is still a large gap between current top detectors and human performance. To narrow this gap, we propose a network fusion architecture, which consists of a multispectral proposal network to generate pedestrian proposals, and a subsequent multispectral classification network to distinguish pedestrian instances from hard negatives. The unified network is learned by jointly optimizing pedestrian detection and semantic segmentation tasks. The final detections are obtained by integrating the outputs from different modalities as well as the two stages. The approach significantly outperforms state-of-the-art methods on the KAIST dataset while remain fast. Additionally, we contribute a sanitized version of training annotations for the KAIST dataset, and examine the effects caused by different kinds of annotation errors. Future research of this problem will benefit from the sanitized version which eliminates the interference of annotation errors.