# Multimodal emotion recognition via adaptive high-order transforme network

## TL;DR

This paper introduces a new method for emotion recognition using multiple data types, which improves accuracy by reducing redundancy and enhancing feature learning.

## Contribution

The novel contribution is the Adaptive High-order Transformer Network (AHOT) with AST and CMFF blocks for better multimodal feature fusion.

## Key findings

- AHOT improves emotion recognition accuracy on IEMOCAP and CMU-MOSEI datasets.
- Ablation studies confirm the effectiveness of the proposed modules in AHOT.

## Abstract

Multimodal emotion recognition leverages multiple modalities to capture emotional cues more comprehensively, thereby improving the accuracy and robustness of emotion recognition. From the perspective of multimodal data and feature learning, reducing information redundancy in multimodal data and enhancing the discriminability of deep feature co-learning can effectively boost recognition performance. Based on this, this paper proposes a multimodal emotion recognition method based on an Adaptive High-order Transformer Network (AHOT). This method constructs Adaptive Selection Transformer block (AST) and Cross-modal Feature Fusion block (CMFF) for each modality branch, aiming to fully capture non-redundant feature representations from each modality and the interactions between modalities. In addition, a sparse high-order feature learning module is designed to enable the learning of highly discriminative high-order features across modalities. Experimental results on two multimodal emotion recognition datasets (IEMOCAP and CMU-MOSEI) demonstrate that, compared with several related methods, the proposed AHOT effectively improves emotion recognition accuracy. Moreover, ablation studies and parameter analyses further validate the effectiveness of AHOT.