MMRel: Benchmarking Relation Understanding in Multi-Modal Large Language Models

TL;DR

This paper introduces MMRel, a large-scale, high-quality benchmark dataset designed to evaluate and improve relation understanding in Multi-modal Large Language Models, addressing current limitations in inter-object relation comprehension.

Contribution

The paper presents MMRel, a comprehensive benchmark with diverse, high-quality relation data, and demonstrates its effectiveness in evaluating and enhancing MLLMs' relation understanding capabilities.

Findings

MMRel improves evaluation accuracy for MLLMs on relation tasks.

Fine-tuning MLLMs with MMRel enhances their relation comprehension.

Extensive experiments validate MMRel's utility across 28 MLLMs.

Abstract

Though Multi-modal Large Language Models (MLLMs) have recently achieved significant progress, they often struggle to understand diverse and complicated inter-object relations. Specifically, the lack of large-scale and high-quality relation data has greatly hindered the progress of MLLMs in various vision-language perception tasks. We attempt to address this challenge by contributing the Multi-Modal Relation Understanding benchmark (MMRel), which features large-scale, high-quality, and diverse data on inter-object relations. MMRel has three distinctive attributes: (i) it contains 22,500 question-answer pairs spanning three distinct domains and around 400 relations, ensuring both scale and diversity; (ii) it provides manually verified, high-quality labels to ensure exceptional annotation accuracy; and (iii) it includes adversarial cases with highly unusual relations, offering a challenging setting for evaluating relation hallucination. These features make MMRel ideal for evaluating MLLMs on relation understanding, as well as for fine-tuning MLLMs to enhance relation comprehension capability. Extensive experiments on 28 MLLMs demonstrate the effectiveness of MMRel in both evaluating and enhancing MLLMs' relation understanding, and the accompanying analyses provide insights for future research. The benchmark has been made publicly available at: https://niejiahao1998.github.io/MMRel