AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian Detection

TL;DR

This paper introduces AMFD, a novel distillation framework that leverages original multi-modal features for improved multispectral pedestrian detection, reducing inference time and enhancing detection accuracy.

Contribution

The proposed AMFD framework fully utilizes teacher network features with a new Modal Extraction Alignment module, enabling effective fusion without extra fusion modules.

Findings

Outperforms state-of-the-art methods in reducing Miss Rate.

Improves mean Average Precision on multiple datasets.

Reduces inference time compared to double-stream networks.

Abstract

Multispectral pedestrian detection has been shown to be effective in improving performance within complex illumination scenarios. However, prevalent double-stream networks in multispectral detection employ two separate feature extraction branches for multi-modal data, leading to nearly double the inference time compared to single-stream networks utilizing only one feature extraction branch. This increased inference time has hindered the widespread employment of multispectral pedestrian detection in embedded devices for autonomous systems. To address this limitation, various knowledge distillation methods have been proposed. However, traditional distillation methods focus only on the fusion features and ignore the large amount of information in the original multi-modal features, thereby restricting the student network's performance. To tackle the challenge, we introduce the Adaptive Modal Fusion Distillation (AMFD) framework, which can fully utilize the original modal features of the teacher network. Specifically, a Modal Extraction Alignment (MEA) module is utilized to derive learning weights for student networks, integrating focal and global attention mechanisms. This methodology enables the student network to acquire optimal fusion strategies independent from that of teacher network without necessitating an additional feature fusion module. Furthermore, we present the SMOD dataset, a well-aligned challenging multispectral dataset for detection. Extensive experiments on the challenging KAIST, LLVIP and SMOD datasets are conducted to validate the effectiveness of AMFD. The results demonstrate that our method outperforms existing state-of-the-art methods in both reducing log-average Miss Rate and improving mean Average Precision. The code is available at https://github.com/bigD233/AMFD.git.