Partitioner Guided Modal Learning Framework

TL;DR

This paper introduces PgM, a novel framework that effectively separates and learns uni-modal and paired-modal features in multimodal learning, improving flexibility and transferability across diverse tasks.

Contribution

The paper proposes a partitioner-guided framework that segments modal representations into uni-modal and paired-modal features, enabling more thorough and adaptable multimodal learning.

Findings

PgM improves performance across four multimodal tasks.

The framework enhances transferability to existing models.

Visualization reveals distinct contributions of features.

Abstract

Multimodal learning benefits from multiple modal information, and each learned modal representations can be divided into uni-modal that can be learned from uni-modal training and paired-modal features that can be learned from cross-modal interaction. Building on this perspective, we propose a partitioner-guided modal learning framework, PgM, which consists of the modal partitioner, uni-modal learner, paired-modal learner, and uni-paired modal decoder. Modal partitioner segments the learned modal representation into uni-modal and paired-modal features. Modal learner incorporates two dedicated components for uni-modal and paired-modal learning. Uni-paired modal decoder reconstructs modal representation based on uni-modal and paired-modal features. PgM offers three key benefits: 1) thorough learning of uni-modal and paired-modal features, 2) flexible distribution adjustment for uni-modal and paired-modal representations to suit diverse downstream tasks, and 3) different learning rates across modalities and partitions. Extensive experiments demonstrate the effectiveness of PgM across four multimodal tasks and further highlight its transferability to existing models. Additionally, we visualize the distribution of uni-modal and paired-modal features across modalities and tasks, offering insights into their respective contributions.