Augmenting Intra-Modal Understanding in MLLMs for Robust Multimodal Keyphrase Generation

TL;DR

This paper introduces AimKP, a framework that enhances intra-modal understanding in multimodal large language models to improve robustness and accuracy in keyphrase generation from noisy, incomplete, or misaligned multimedia data.

Contribution

AimKP explicitly reinforces intra-modal semantic learning in MLLMs through progressive masking and gradient filtering, addressing modality bias and improving robustness.

Findings

Achieves superior keyphrase generation accuracy.

Demonstrates robustness across noisy and incomplete data.

Outperforms existing methods in diverse scenarios.

Abstract

Multimodal keyphrase generation (MKP) aims to extract a concise set of keyphrases that capture the essential meaning of paired image-text inputs, enabling structured understanding, indexing, and retrieval of multimedia data across the web and social platforms. Success in this task demands effectively bridging the semantic gap between heterogeneous modalities. While multimodal large language models (MLLMs) achieve superior cross-modal understanding by leveraging massive pretraining on image-text corpora, we observe that they often struggle with modality bias and fine-grained intra-modal feature extraction. This oversight leads to a lack of robustness in real-world scenarios where multimedia data is noisy, along with incomplete or misaligned modalities. To address this problem, we propose AimKP, a novel framework that explicitly reinforces intra-modal semantic learning in MLLMs while preserving cross-modal alignment. AimKP incorporates two core innovations: (i) Progressive Modality Masking, which forces fine-grained feature extraction from corrupted inputs by progressively masking modality information during training; (ii) Gradient-based Filtering, that identifies and discards noisy samples, preventing them from corrupting the model's core cross-modal learning. Extensive experiments validate AimKP's effectiveness in multimodal keyphrase generation and its robustness across different scenarios.