RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation

TL;DR

This paper introduces RPTS, a tree-structured metric for evaluating the reasoning process in multimodal models, and presents RPTS-Eval, a benchmark to assess and analyze their reasoning capabilities.

Contribution

The paper proposes a novel tree-based reasoning process scoring method and constructs a new benchmark for more faithful evaluation of multimodal reasoning in LVLMs.

Findings

LVLMs show limitations in multimodal reasoning accuracy.

RPTS effectively pinpoints reasoning failures and strengths.

Benchmark reveals differences between open-source and commercial LVLMs.

Abstract

Large Vision-Language Models (LVLMs) excel in multimodal reasoning and have shown impressive performance on various multimodal benchmarks. However, most of these benchmarks evaluate models primarily through multiple-choice or short-answer formats, which do not take the reasoning process into account. Although some benchmarks assess the reasoning process, their methods are often overly simplistic and only examine reasoning when answers are incorrect. This approach overlooks scenarios where flawed reasoning leads to correct answers. In addition, these benchmarks do not consider the impact of intermodal relationships on reasoning. To address this issue, we propose the Reasoning Process Tree Score (RPTS), a tree structure-based metric to assess reasoning processes. Specifically, we organize the reasoning steps into a reasoning tree and leverage its hierarchical information to assign weighted faithfulness scores to each reasoning step. By dynamically adjusting these weights, RPTS not only evaluates the overall correctness of the reasoning, but also pinpoints where the model fails in the reasoning. To validate RPTS in real-world multimodal scenarios, we construct a new benchmark, RPTS-Eval, comprising 374 images and 390 reasoning instances. Each instance includes reliable visual-textual clues that serve as leaf nodes of the reasoning tree. Furthermore, we define three types of intermodal relationships to investigate how intermodal interactions influence the reasoning process. We evaluated representative LVLMs (e.g., GPT4o, Llava-Next), uncovering their limitations in multimodal reasoning and highlighting the differences between open-source and closed-source commercial LVLMs. We believe that this benchmark will contribute to the advancement of research in the field of multimodal reasoning.