MIMIC-IT: Multi-Modal In-Context Instruction Tuning

TL;DR

MIMIC-IT introduces a large, diverse dataset of 2.8 million multimodal instruction-response pairs with multi-modal context, enabling improved training and evaluation of vision-language models like Otter for perception, reasoning, and planning tasks.

Contribution

The paper presents MIMIC-IT, a novel large-scale multimodal instruction-response dataset with multi-modal context, and demonstrates its effectiveness by training Otter, a VLM that excels in perception and reasoning.

Findings

Otter shows strong performance on vision-language benchmarks.

MIMIC-IT dataset enhances multi-modal perception and reasoning.

Human evaluation confirms alignment with user intentions.

Abstract

High-quality instructions and responses are essential for the zero-shot performance of large language models on interactive natural language tasks. For interactive vision-language tasks involving intricate visual scenes, a large quantity of diverse and creative instruction-response pairs should be imperative to tune vision-language models (VLMs). Nevertheless, the current availability of vision-language instruction-response pairs in terms of quantity, diversity, and creativity remains limited, posing challenges to the generalization of interactive VLMs. Here we present MultI-Modal In-Context Instruction Tuning (MIMIC-IT), a dataset comprising 2.8 million multimodal instruction-response pairs, with 2.2 million unique instructions derived from images and videos. Each pair is accompanied by multi-modal in-context information, forming conversational contexts aimed at empowering VLMs in perception, reasoning, and planning. The instruction-response collection process, dubbed as Syphus, is scaled using an automatic annotation pipeline that combines human expertise with GPT's capabilities. Using the MIMIC-IT dataset, we train a large VLM named Otter. Based on extensive evaluations conducted on vision-language benchmarks, it has been observed that Otter demonstrates remarkable proficiency in multi-modal perception, reasoning, and in-context learning. Human evaluation reveals it effectively aligns with the user's intentions. We release the MIMIC-IT dataset, instruction-response collection pipeline, benchmarks, and the Otter model.