Generalized Bilinear Deep Convolutional Neural Networks for Multimodal Biometric Identification

TL;DR

This paper introduces a multimodal biometric identification system using modality-specific CNNs fused at the feature level, employing generalized compact bilinear fusion for improved accuracy and efficiency across multiple challenging datasets.

Contribution

It proposes a novel generalized compact bilinear fusion method for multimodal CNNs, enhancing biometric identification performance with reduced parameters.

Findings

Significant performance improvement over unimodal systems.

Effective fusion at the fully-connected layer without performance loss.

Robust results across three challenging biometric datasets.

Abstract

In this paper, we propose to employ a bank of modality-dedicated Convolutional Neural Networks (CNNs), fuse, train, and optimize them together for person classification tasks. A modality-dedicated CNN is used for each modality to extract modality-specific features. We demonstrate that, rather than spatial fusion at the convolutional layers, the fusion can be performed on the outputs of the fully-connected layers of the modality-specific CNNs without any loss of performance and with significant reduction in the number of parameters. We show that, using multiple CNNs with multimodal fusion at the feature-level, we significantly outperform systems that use unimodal representation. We study weighted feature, bilinear, and compact bilinear feature-level fusion algorithms for multimodal biometric person identification. Finally, We propose generalized compact bilinear fusion algorithm to deploy both the weighted feature fusion and compact bilinear schemes. We provide the results for the proposed algorithms on three challenging databases: CMU Multi-PIE, BioCop, and BIOMDATA.