On the Performance of Multimodal Language Models

TL;DR

This paper analyzes various multimodal instruction tuning methods for large language models integrated with vision, benchmarking their performance across diverse tasks and highlighting current limitations in dataset diversity and factual accuracy.

Contribution

It provides a comprehensive comparison of multimodal instruction tuning approaches and offers insights into architectural choices and existing challenges.

Findings

Multimodal models show promising zero-shot capabilities across tasks.

Current approaches lack diverse instruction datasets for better generalization.

Factuality issues remain a concern in multimodal response generation.

Abstract

Instruction-tuned large language models (LLMs) have demonstrated promising zero-shot generalization capabilities across various downstream tasks. Recent research has introduced multimodal capabilities to LLMs by integrating independently pretrained vision encoders through model grafting. These multimodal variants undergo instruction tuning, similar to LLMs, enabling effective zero-shot generalization for multimodal tasks. This study conducts a comparative analysis of different multimodal instruction tuning approaches and evaluates their performance across a range of tasks, including complex reasoning, conversation, image captioning, multiple-choice questions (MCQs), and binary classification. Through rigorous benchmarking and ablation experiments, we reveal key insights for guiding architectural choices when incorporating multimodal capabilities into LLMs. However, current approaches have limitations; they do not sufficiently address the need for a diverse multimodal instruction dataset, which is crucial for enhancing task generalization. Additionally, they overlook issues related to truthfulness and factuality when generating responses. These findings illuminate current methodological constraints in adapting language models for image comprehension and provide valuable guidance for researchers and practitioners seeking to harness multimodal versions of LLMs.