Multimodal Chain-of-Thought Reasoning in Language Models

TL;DR

This paper introduces Multimodal-CoT, a framework that combines text and images for reasoning in language models, improving accuracy and efficiency on science and visual question-answering benchmarks.

Contribution

It extends chain-of-thought reasoning to multimodal inputs, enabling better rationales and answer inference in language models with a two-stage process.

Findings

Achieves state-of-the-art performance on ScienceQA with under 1B parameters.

Mitigates hallucination and improves convergence speed.

Effective in multimodal reasoning tasks.

Abstract

Large language models (LLMs) have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer. However, existing CoT studies have primarily focused on the language modality. We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information. Experimental results on ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed approach. With Multimodal-CoT, our model under 1 billion parameters achieves state-of-the-art performance on the ScienceQA benchmark. Our analysis indicates that Multimodal-CoT offers the advantages of mitigating hallucination and enhancing convergence speed. Code is publicly available at https://github.com/amazon-science/mm-cot.