On Domain-Adaptive Post-Training for Multimodal Large Language Models

TL;DR

This paper presents a systematic approach for domain adaptation of multimodal large language models through post-training, emphasizing data synthesis, training strategies, and extensive domain-specific evaluations.

Contribution

It introduces a generate-then-filter data synthesis pipeline, demonstrates the effectiveness of single-stage training for domain adaptation, and provides comprehensive evaluations across multiple high-impact domains.

Findings

Generated domain-specific data outperforms manual and closed-source methods.

Single-stage training surpasses two-stage approaches for domain adaptation.

Extensive experiments validate improved performance in biomedicine, food, and remote sensing.

Abstract

Adapting general multimodal large language models (MLLMs) to specific domains, such as scientific and industrial fields, is highly significant in promoting their practical applications. This paper systematically investigates domain adaptation of MLLMs via post-training, focusing on data synthesis, training pipeline, and task evaluation. (1) Data Synthesis: Using only open-source models, we develop a generate-then-filter pipeline that curates diverse visual instruction tasks based on domain-specific image-caption pairs. The resulting data surpass the data synthesized by manual rules or strong closed-source models in enhancing domain-specific performance. (2) Training Pipeline: Unlike general MLLMs that typically adopt a two-stage training paradigm, we find that a single-stage approach is more effective for domain adaptation. (3) Task Evaluation: We conduct extensive experiments in high-impact domains such as biomedicine, food, and remote sensing, by post-training a variety of MLLMs and then evaluating MLLM performance on various domain-specific tasks. Finally, we fully open-source our models, code, and data to encourage future research in this area.