Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Images

TL;DR

This paper introduces STAR-64K, a large multi-modal dataset and a two-stage training framework that significantly improves smaller models' reasoning capabilities on multi-modal relational knowledge tasks.

Contribution

The paper presents a novel data synthesis engine and a two-stage training framework for enhancing multi-modal reasoning in large language models.

Findings

Smaller models outperform GPT-4o in STAR tasks after training.

The STAR-64K dataset enables effective multi-modal relational reasoning.

The proposed framework improves reasoning capabilities with scalable data and training methods.

Abstract

Understanding and reasoning with abstractive information from the visual modality presents significant challenges for current multi-modal large language models (MLLMs). Among the various forms of abstractive information, Multi-Modal Relational Knowledge (MMRK), which represents abstract relational structures between multi-modal entities using node-edge formats, remains largely under-explored. In particular, STructured and Abstractive Reasoning (STAR) on such data has received little attention from the research community. To bridge the dual gaps in large-scale high-quality data and capability enhancement methodologies, this paper makes the following key contributions: (i). An automatic STAR data engine capable of synthesizing images with MMRK to build multi-modal instruction data with reliable chain-of-thought thinking for various STAR tasks and (ii). A comprehsive two-stage capability enhancement training framework, accompanied by a suite of evaluation protocols tailored to different STAR tasks. Based upon these contributions, we introduce STAR-64K, a dataset comprising 64K high-quality multi-modal instruction samples, and conduct experiments across 5 open-source MLLMs. Experimental results show that our two-stage enhancement framework enables smaller 3B/7B models to significantly outperform GPT-4o in STAR. Additionally, we provide in-depth analysis regarding the effectiveness of various designs, data transferability, and scalability.