Multimodal Classification and Out-of-distribution Detection for Multimodal Intent Understanding

TL;DR

This paper introduces MIntOOD, a novel multimodal intent understanding method that enhances in-distribution classification and out-of-distribution detection by dynamic feature fusion and multimodal representation learning, achieving state-of-the-art results.

Contribution

The paper proposes a weighted feature fusion network and a dual-perspective representation learning approach for improved multimodal intent classification and OOD detection.

Findings

Significant improvement in OOD detection AUROC scores (3-10%)

State-of-the-art accuracy in ID classification

Effective multimodal representation learning from coarse and fine-grained views

Abstract

Multimodal intent understanding is a significant research area that requires effective leveraging of multiple modalities to analyze human language. Existing methods face two main challenges in this domain. Firstly, they have limitations in capturing the nuanced and high-level semantics underlying complex in-distribution (ID) multimodal intents. Secondly, they exhibit poor generalization when confronted with unseen out-of-distribution (OOD) data in real-world scenarios. To address these issues, we propose a novel method for both ID classification and OOD detection (MIntOOD). We first introduce a weighted feature fusion network that models multimodal representations. This network dynamically learns the importance of each modality, adapting to multimodal contexts. To develop discriminative representations for both tasks, we synthesize pseudo-OOD data from convex combinations of ID data and engage in multimodal representation learning from both coarse-grained and fine-grained perspectives. The coarse-grained perspective focuses on distinguishing between ID and OOD binary classes, while the fine-grained perspective not only enhances the discrimination between different ID classes but also captures instance-level interactions between ID and OOD samples, promoting proximity among similar instances and separation from dissimilar ones. We establish baselines for three multimodal intent datasets and build an OOD benchmark. Extensive experiments on these datasets demonstrate that our method significantly improves OOD detection performance with a 3~10% increase in AUROC scores while achieving new state-of-the-art results in ID classification. Data and codes are available at https://github.com/thuiar/MIntOOD.