Corvid: Improving Multimodal Large Language Models Towards Chain-of-Thought Reasoning

TL;DR

Corvid is a multimodal large language model enhanced with chain-of-thought reasoning, utilizing a hybrid vision encoder, specialized training datasets, and inference strategies to improve complex reasoning tasks like mathematics and science problem-solving.

Contribution

The paper introduces Corvid, a novel MLLM with improved CoT reasoning through a hybrid vision encoder, a new instruction dataset, and a two-stage training and inference approach.

Findings

Outperforms existing MLLMs in reasoning tasks

Excels in mathematical and science problem-solving

Effective self-verification during inference

Abstract

Recent advancements in multimodal large language models (MLLMs) have demonstrated exceptional performance in multimodal perception and understanding. However, leading open-source MLLMs exhibit significant limitations in complex and structured reasoning, particularly in tasks requiring deep reasoning for decision-making and problem-solving. In this work, we present Corvid, an MLLM with enhanced chain-of-thought (CoT) reasoning capabilities. Architecturally, Corvid incorporates a hybrid vision encoder for informative visual representation and a meticulously designed connector (GateMixer) to facilitate cross-modal alignment. To enhance Corvid's CoT reasoning capabilities, we introduce MCoT-Instruct-287K, a high-quality multimodal CoT instruction-following dataset, refined and standardized from diverse public reasoning sources. Leveraging this dataset, we fine-tune Corvid with a two-stage CoT-formatted training approach to progressively enhance its step-by-step reasoning abilities. Furthermore, we propose an effective inference-time scaling strategy that enables Corvid to mitigate over-reasoning and under-reasoning through self-verification. Extensive experiments demonstrate that Corvid outperforms existing o1-like MLLMs and state-of-the-art MLLMs with similar parameter scales, with notable strengths in mathematical reasoning and science problem-solving. Project page: https://mm-vl.github.io/corvid.