MINT: Multimodal Instruction Tuning with Multimodal Interaction Grouping

TL;DR

MINT introduces a task-grouping strategy based on multimodal interactions that improves instruction tuning performance by enhancing model transferability and reducing task interference.

Contribution

The paper proposes a novel task grouping method for multimodal instruction tuning based on interaction types, outperforming existing grouping strategies.

Findings

MINT significantly outperforms baseline task grouping methods.

Grouping by interaction types improves transferability and reduces interference.

Effective balance between generalization and specialization achieved.

Abstract

Recent advances in multimodal foundation models have achieved state-of-the-art performance across a range of tasks. These breakthroughs are largely driven by new pre-training paradigms that leverage large-scale, unlabeled multimodal data, followed by instruction fine-tuning on curated labeled datasets and high-quality prompts. While there is growing interest in scaling instruction fine-tuning to ever-larger datasets in both quantity and scale, our findings reveal that simply increasing the number of instruction-tuning tasks does not consistently yield better performance. Instead, we observe that grouping tasks by the common interactions across modalities, such as discovering redundant shared information, prioritizing modality selection with unique information, or requiring synergistic fusion to discover new information from both modalities, encourages the models to learn transferrable skills within a group while suppressing interference from mismatched tasks. To this end, we introduce MINT, a simple yet surprisingly effective task-grouping strategy based on the type of multimodal interaction. We demonstrate that the proposed method greatly outperforms existing task grouping baselines for multimodal instruction tuning, striking an effective balance between generalization and specialization.