Emotion and Intention Guided Multi-Modal Learning for Sticker Response Selection

TL;DR

This paper introduces a novel multi-modal learning framework that jointly models emotion and intention to improve sticker response selection in online dialogue, addressing limitations of previous isolated modeling approaches.

Contribution

It proposes the first joint modeling framework for emotion and intention in multi-modal learning, incorporating dual-level contrastive alignment and a progressive fusion module.

Findings

Outperforms state-of-the-art methods on two public datasets

Achieves higher accuracy in sticker response selection

Demonstrates effective integration of emotional and intentional cues

Abstract

Stickers are widely used in online communication to convey emotions and implicit intentions. The Sticker Response Selection (SRS) task aims to select the most contextually appropriate sticker based on the dialogue. However, existing methods typically rely on semantic matching and model emotional and intentional cues separately, which can lead to mismatches when emotions and intentions are misaligned. To address this issue, we propose Emotion and Intention Guided Multi-Modal Learning (EIGML). This framework is the first to jointly model emotion and intention, effectively reducing the bias caused by isolated modeling and significantly improving selection accuracy. Specifically, we introduce Dual-Level Contrastive Framework to perform both intra-modality and inter-modality alignment, ensuring consistent representation of emotional and intentional features within and across modalities. In addition, we design an Intention-Emotion Guided Multi-Modal Fusion module that integrates emotional and intentional information progressively through three components: Emotion-Guided Intention Knowledge Selection, Intention-Emotion Guided Attention Fusion, and Similarity-Adjusted Matching Mechanism. This design injects rich, effective information into the model and enables a deeper understanding of the dialogue, ultimately enhancing sticker selection performance. Experimental results on two public SRS datasets show that EIGML consistently outperforms state-of-the-art baselines, achieving higher accuracy and a better understanding of emotional and intentional features. Code is provided in the supplementary materials.