T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering

TL;DR

T-SciQ introduces a novel method that leverages large language model signals to generate high-quality reasoning rationales, enabling smaller models to excel in complex multimodal science question answering tasks with state-of-the-art accuracy.

Contribution

The paper presents T-SciQ, a new approach that uses LLM signals to teach reasoning, reducing reliance on costly human annotations and improving performance on science QA benchmarks.

Findings

Achieves 96.18% accuracy on ScienceQA benchmark.

Outperforms fine-tuned baselines by 4.5%.

Demonstrates effective teaching of reasoning in complex modalities.

Abstract

Large Language Models (LLMs) have recently demonstrated exceptional performance in various Natural Language Processing (NLP) tasks. They have also shown the ability to perform chain-of-thought (CoT) reasoning to solve complex problems. Recent studies have explored CoT reasoning in complex multimodal scenarios, such as the science question answering task, by fine-tuning multimodal models with high-quality human-annotated CoT rationales. However, collecting high-quality COT rationales is usually time-consuming and costly. Besides, the annotated rationales are hardly accurate due to the external essential information missed. To address these issues, we propose a novel method termed T-SciQ that aims at teaching science question answering with LLM signals. The T-SciQ approach generates high-quality CoT rationales as teaching signals and is advanced to train much smaller models to perform CoT reasoning in complex modalities. Additionally, we introduce a novel data mixing strategy to produce more effective teaching data samples for simple and complex science question answer problems. Extensive experimental results show that our T-SciQ method achieves a new state-of-the-art performance on the ScienceQA benchmark, with an accuracy of 96.18%. Moreover, our approach outperforms the most powerful fine-tuned baseline by 4.5%. The code is publicly available at https://github.com/T-SciQ/T-SciQ.