Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework

TL;DR

This paper introduces COCO-MMR, a challenging open-ended multimodal reasoning dataset based on COCO, and proposes innovative techniques like multi-hop attention and contrastive learning to improve reasoning capabilities in AI models.

Contribution

The paper presents a new open-ended multimodal reasoning dataset and novel techniques to enhance reasoning performance, addressing limitations of previous datasets and approaches.

Findings

The dataset effectively evaluates multimodal reasoning with open-ended questions.

Proposed techniques improve the reasoning accuracy of models on the dataset.

Extensive experiments validate the effectiveness of the dataset and methods.

Abstract

Multimodal reasoning is a critical component in the pursuit of artificial intelligence systems that exhibit human-like intelligence, especially when tackling complex tasks. While the chain-of-thought (CoT) technique has gained considerable attention, the existing ScienceQA dataset, which focuses on multimodal scientific questions and explanations from elementary and high school textbooks, lacks a comprehensive evaluation of diverse approaches. To address this gap, we present COCO Multi-Modal Reasoning(COCO-MMR) dataset, a novel dataset that encompasses an extensive collection of open-ended questions, rationales, and answers derived from the large object dataset COCO. Unlike previous datasets that rely on multiple-choice questions, our dataset pioneers the use of open-ended questions in the context of multimodal CoT, introducing a more challenging problem that effectively assesses the reasoning capability of CoT models. Through comprehensive evaluations and detailed analyses, we provide valuable insights and propose innovative techniques, including multi-hop cross-modal attention and sentence-level contrastive learning, to enhance the image and text encoders. Extensive experiments demonstrate the efficacy of the proposed dataset and techniques, offering novel perspectives for advancing multimodal reasoning. The data and code are available at \href{https://github.com/weijingxuan/COCO-MMR}{https://github.com/weijingxuan/COCO-MMR}.