Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Model

TL;DR

This paper explores how the number of instruction templates affects multimodal language model training, finding that medium-scale templates yield optimal performance and significant gains over original data.

Contribution

It introduces a programmatic template generator to systematically study the scaling effect of instruction templates on MLM training performance.

Findings

Optimal performance at medium template scale

Up to 10% performance improvement with data augmentation

Best results compared to similar-scale models trained on larger data

Abstract

Current multimodal language model (MLM) training approaches overlook the influence of instruction templates. Previous research deals with this problem by leveraging hand-crafted or model-generated templates, failing to investigate the scaling effect of instruction templates on MLM training. In this work, we propose a programmatic instruction template generator capable of producing over 15K unique instruction templates by filling randomly sampled positional synonyms into weighted sampled meta templates, enabling us to comprehensively explore MLM's performance across various template scales in the training process. Our investigation into scaling instruction templates for MLM training demonstrates that MLM capabilities do not consistently improve with increasing template scale. Instead, optimal performance is achieved at a medium template scale. Models trained with data augmented at the optimal template scale achieve performance gains of up to 10% over those trained on the original data and achieve the best overall performance compared with the similar-scale MLMs tuned on at most 75 times the scale of our augmented dataset. The code will be publicly available at https://github.com/shijian2001/TemplateScaling.