Error-driven Data-efficient Large Multimodal Model Tuning

TL;DR

This paper introduces an error-driven, data-efficient tuning framework for large multimodal models that adapts generic models to new tasks without requiring task-specific training data, significantly improving performance.

Contribution

It proposes a novel error-driven tuning method that leverages a teacher-student setup to identify gaps and retrieve relevant data, enabling efficient adaptation without task-specific samples.

Findings

Achieves an average performance boost of 7.01% across multiple tasks.

Effectively adapts models without task-specific training data.

Demonstrates robustness across different data scales.

Abstract

Large Multimodal Models (LMMs) have demonstrated impressive performance across numerous academic benchmarks. However, fine-tuning still remains essential to achieve satisfactory performance on downstream tasks, while the task-specific tuning samples are usually not readily available or expensive and time-consuming to obtain. To address this, we propose an error-driven data-efficient tuning framework that aims to efficiently adapt generic LMMs to newly emerging tasks without requiring any task-specific training samples. In our approach, a generic LMM, acting as a student model, is first evaluated on a small validation set of the target task, and then a more powerful model, acting as a teacher model, identifies the erroneous steps within the student model's reasoning steps and analyzes its capability gaps from fully addressing the target task. Based on these gaps, targeted training samples are further retrieved from existing task-agnostic datasets to tune the student model and tailor it to the target task. We perform extensive experiments across three different training data scales and seven tasks, demonstrating that our training paradigm significantly and efficiently improves LMM's performance on downstream tasks, achieving an average performance boost of 7.01%.