Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendation

TL;DR

This paper introduces MetaMMF, a meta-learning framework for dynamic multimodal fusion in micro-video recommendation, significantly improving recommendation accuracy by customizing fusion parameters per video.

Contribution

The paper proposes a novel meta-learning-based approach for dynamic multimodal fusion, addressing the limitations of static fusion methods in micro-video recommendation.

Findings

MetaMMF outperforms state-of-the-art models on benchmark datasets.

MetaMMF achieves higher recommendation accuracy with efficient training.

Canonical polyadic decomposition enhances model efficiency without sacrificing performance.

Abstract

Multimodal information (e.g., visual, acoustic, and textual) has been widely used to enhance representation learning for micro-video recommendation. For integrating multimodal information into a joint representation of micro-video, multimodal fusion plays a vital role in the existing micro-video recommendation approaches. However, the static multimodal fusion used in previous studies is insufficient to model the various relationships among multimodal information of different micro-videos. In this paper, we develop a novel meta-learning-based multimodal fusion framework called Meta Multimodal Fusion (MetaMMF), which dynamically assigns parameters to the multimodal fusion function for each micro-video during its representation learning. Specifically, MetaMMF regards the multimodal fusion of each micro-video as an independent task. Based on the meta information extracted from the multimodal features of the input task, MetaMMF parameterizes a neural network as the item-specific fusion function via a meta learner. We perform extensive experiments on three benchmark datasets, demonstrating the significant improvements over several state-of-the-art multimodal recommendation models, like MMGCN, LATTICE, and InvRL. Furthermore, we lighten our model by adopting canonical polyadic decomposition to improve the training efficiency, and validate its effectiveness through experimental results. Codes are available at https://github.com/hanliu95/MetaMMF.