Feature Entanglement-based Quantum Multimodal Fusion Neural Network

TL;DR

This paper introduces a quantum multimodal fusion neural network that balances accuracy, interpretability, and complexity, leveraging quantum computing to improve multimodal data integration.

Contribution

It proposes a novel quantum-based fusion model combining classical and quantum components to reduce complexity and enhance interpretability in multimodal learning.

Findings

Achieves classification accuracy comparable to classical models with fewer parameters

Reduces fusion complexity to linear scale

Demonstrates stability across multimodal image datasets

Abstract

Multimodal learning aims to enhance perceptual and decision-making capabilities by integrating information from diverse sources. However, classical deep learning approaches face a critical trade-off between the high accuracy of black-box feature-level fusion and the interpretability of less outstanding decision-level fusion, alongside the challenges of parameter explosion and complexity. This paper discusses the accuracy-interpretablity-complexity dilemma under the quantum computation framework and propose a feature entanglement-based quantum multimodal fusion neural network. The model is composed of three core components: a classical feed-forward module for unimodal processing, an interpretable quantum fusion block, and a quantum convolutional neural network (QCNN) for deep feature extraction. By leveraging the strong expressive power of quantum, we have reduced the complexity of multimodal fusion and post-processing to linear, and the fusion process also possesses the interpretability of decision-level fusion. The simulation results demonstrate that our model achieves classification accuracy comparable to classical networks with dozens of times of parameters, exhibiting notable stability and performance across multimodal image datasets.