Neural Dependency Coding inspired Multimodal Fusion

TL;DR

This paper introduces a neural dependency coding approach inspired by neuroscience to improve multimodal fusion, demonstrating consistent performance gains in sentiment analysis tasks.

Contribution

It proposes a novel synergy maximizing loss function for neural multimodal fusion, inspired by biological multisensory integration.

Findings

Performance improvements on CMU-MOSI and CMU-MOSEI datasets

Enhanced multimodal sentiment analysis accuracy

Effective synergy maximization in neural fusion models

Abstract

Information integration from different modalities is an active area of research. Human beings and, in general, biological neural systems are quite adept at using a multitude of signals from different sensory perceptive fields to interact with the environment and each other. Recent work in deep fusion models via neural networks has led to substantial improvements over unimodal approaches in areas like speech recognition, emotion recognition and analysis, captioning and image description. However, such research has mostly focused on architectural changes allowing for fusion of different modalities while keeping the model complexity manageable. Inspired by recent neuroscience ideas about multisensory integration and processing, we investigate the effect of synergy maximizing loss functions. Experiments on multimodal sentiment analysis tasks: CMU-MOSI and CMU-MOSEI with different models show that our approach provides a consistent performance boost.