Multi-attention Recurrent Network for Human Communication Comprehension

TL;DR

This paper introduces MARN, a neural network architecture that effectively models interactions between language, vision, and acoustic modalities over time to understand human communication, achieving state-of-the-art results.

Contribution

The paper proposes the Multi-attention Recurrent Network (MARN), a novel neural architecture that captures multimodal interactions through time using multi-attention and hybrid memory components.

Findings

MARN achieves state-of-the-art performance on six multimodal datasets.

The model effectively captures interactions between modalities over time.

MARN outperforms existing methods in sentiment, emotion, and trait recognition.

Abstract

Human face-to-face communication is a complex multimodal signal. We use words (language modality), gestures (vision modality) and changes in tone (acoustic modality) to convey our intentions. Humans easily process and understand face-to-face communication, however, comprehending this form of communication remains a significant challenge for Artificial Intelligence (AI). AI must understand each modality and the interactions between them that shape human communication. In this paper, we present a novel neural architecture for understanding human communication called the Multi-attention Recurrent Network (MARN). The main strength of our model comes from discovering interactions between modalities through time using a neural component called the Multi-attention Block (MAB) and storing them in the hybrid memory of a recurrent component called the Long-short Term Hybrid Memory (LSTHM). We perform extensive comparisons on six publicly available datasets for multimodal sentiment analysis, speaker trait recognition and emotion recognition. MARN shows state-of-the-art performance on all the datasets.