MM-PhyQA: Multimodal Physics Question-Answering With Multi-Image CoT Prompting

TL;DR

This paper introduces MM-PhyQA, a high school-level multimodal physics question dataset, and proposes MI-CoT prompting, demonstrating improved multimodal reasoning performance with a maximum accuracy of 71.65%.

Contribution

The paper creates a new multimodal physics dataset and introduces MI-CoT prompting, enhancing LLMs' multi-image reasoning capabilities.

Findings

LLaVA-1.5 with MI-CoT achieved the highest accuracy of 71.65%.

Multimodal prompting significantly improves physics reasoning performance.

Fine-tuned models outperform zero-shot GPT-4 on the dataset.

Abstract

While Large Language Models (LLMs) can achieve human-level performance in various tasks, they continue to face challenges when it comes to effectively tackling multi-step physics reasoning tasks. To identify the shortcomings of existing models and facilitate further research in this area, we curated a novel dataset, MM-PhyQA, which comprises well-constructed, high schoollevel multimodal physics problems. By evaluating the performance of contemporary LLMs that are publicly available, both with and without the incorporation of multimodal elements in these problems, we aim to shed light on their capabilities. For generating answers for questions consisting of multimodal input (in this case, images and text) we employed Zero-shot prediction using GPT-4 and utilized LLaVA (LLaVA and LLaVA-1.5), the latter of which were fine-tuned on our dataset. For evaluating the performance of LLMs consisting solely of textual input, we tested the performance of the base and fine-tuned versions of the Mistral-7B and LLaMA2-7b models. We also showcased the performance of the novel Multi-Image Chain-of-Thought (MI-CoT) Prompting technique, which when used to train LLaVA-1.5 13b yielded the best results when tested on our dataset, with superior scores in most metrics and the highest accuracy of 71.65% on the test set.