# An innovative 3D attention mechanism for multi-label emotion classification

## TL;DR

This paper introduces a new 3D attention mechanism for emotion classification that improves performance on complex emotion datasets.

## Contribution

The novel Commander Attention mechanism enhances learning by using 3D attention planes and emotional features like polarity and intensity.

## Key findings

- The 3-CA mechanism outperforms existing methods in classifying 28 emotion categories.
- It achieves improved results in over 85.7% of emotion categories compared to state-of-the-art approaches.
- The method uses multi-task learning and a learnable mixing weight parameter for predictions.

## Abstract

In recent years, integrating pre-trained models with attention mechanisms has become a prevalent approach in multi-label emotion classification tasks. However, most researchers focus on modifying the attention network structure or substituting it with larger pre-trained models, often overlooking the enhancement of the attention mechanism’s learning capabilities. This paper introduces a novel attention mechanism, exemplified by the GoEmotions dataset, which encompasses 28 emotion categories, the most complex set to date. We devised distinct attention layers for each emotion label, conceptualized as separate two-dimensional planes, each containing reviews pertinent to the respective label. These planes, when stacked, form a three-dimensional cube. A multi-head attention mechanism is subsequently employed to establish connections between these planes. Additionally, we incorporated the dimensions of emotional polarity and intensity, absent in the original dataset, and defined this feature as “Commander”. The Commander functions as follows: Within each plane, it adjusts attention weight parameters among reviews via linear and non-linear transformations. Across the cube, the 28 Commanders, representing the 28 emotion labels, are determined by averaging the emotional polarity and intensity for each label. Utilizing a multi-task learning approach, we independently trained and stored predictions for each emotion. The Commanders then combine these prediction results linearly through a learnable mixing weight parameter, which is integrated into the input of the multi-head attention mechanism. We term this mechanism, which operates in a 3D attention space and guides attention learning via indicative features, as Commander Attention. When coupled with the XLNet pre-trained model for fine-tuning in downstream tasks, 3-CA outperforms the original method in the classification of all emotion categories and achieves a minimum improvement in the classification of over 85.7% of emotion categories compared to various current state-of-the-art methods. We have made the relevant core code available at https://github.com/FamerKing/Commander-Attention and will continue to update it with the complete implementation in the future.