Generalized Multimodal Fusion via Poisson-Nernst-Planck Equation

TL;DR

This paper introduces a novel multimodal fusion method based on the Poisson-Nernst-Planck equation, improving feature extraction, data integrity, and adaptability across tasks with fewer resources, and achieving state-of-the-art performance.

Contribution

The paper presents a generalized multimodal fusion approach using PNP equations, integrating physics-inspired modeling to enhance feature disentanglement and efficiency.

Findings

Achieves near state-of-the-art accuracy with fewer parameters.

Reduces mutual information and entropy in feature representations.

Surpasses SOTA results when combined with advanced fusion methods.

Abstract

Previous studies have highlighted significant advancements in multimodal fusion. Nevertheless, such methods often encounter challenges regarding the efficacy of feature extraction, data integrity, consistency of feature dimensions, and adaptability across various downstream tasks. This paper proposes a generalized multimodal fusion method (GMF) via the Poisson-Nernst-Planck (PNP) equation, which adeptly addresses the aforementioned issues. Theoretically, the optimization objective for traditional multimodal tasks is formulated and redefined by integrating information entropy and the flow of gradient backward step. Leveraging these theoretical insights, the PNP equation is applied to feature fusion, rethinking multimodal features through the framework of charged particles in physics and controlling their movement through dissociation, concentration, and reconstruction. Building on these theoretical foundations, GMF disassociated features which extracted by the unimodal feature extractor into modality-specific and modality-invariant subspaces, thereby reducing mutual information and subsequently lowering the entropy of downstream tasks. The identifiability of the feature's origin enables our approach to function independently as a frontend, seamlessly integrated with a simple concatenation backend, or serve as a prerequisite for other modules. Experimental results on multiple downstream tasks show that the proposed GMF achieves performance close to the state-of-the-art (SOTA) accuracy while utilizing fewer parameters and computational resources. Furthermore, by integrating GMF with advanced fusion methods, we surpass the SOTA results.