MMFCTUB: Multi-Modal Financial Credit Table Understanding Benchmark

TL;DR

This paper introduces MMFCTUB, a comprehensive benchmark with over 7,600 samples for evaluating multi-modal language models on financial credit table understanding, addressing data, annotation, and evaluation challenges.

Contribution

The paper presents MMFCTUB, a new benchmark for financial credit table understanding that uses a minimally supervised pipeline and diverse evaluation strategies.

Findings

MLLMs show strengths in cross-table structure perception.

MLLMs have limitations in domain knowledge utilization.

Benchmark reveals gaps in numerical calculation capabilities.

Abstract

The advent of multi-modal language models (MLLMs) has spurred research into their application across various table understanding tasks. However, their performance in credit table understanding (CTU) for financial credit review remains largely unexplored due to the following barriers: low data consistency, high annotation costs stemming from domain-specific knowledge and complex calculations, and evaluation paradigm gaps between benchmark and real-world scenarios. To address these challenges, we introduce MMFCTUB (Multi-Modal Financial Credit Table Understanding Benchmark), a practical benchmark, encompassing more than 7,600 high quality CTU samples across 5 table types. MMFCTUB employ a minimally supervised pipeline that adheres to inter-table constraints and maintains data distributions consistency. The benchmark leverages capacity-driven questions and mask-and-recovery strategy to evaluate models' cross-table structure perception, domain knowledge utilization, and numerical calculation capabilities. Utilizing MMFCTUB, we conduct comprehensive evaluations of both proprietary and open-source MLLMs, revealing their strengths and limitations in CTU tasks. MMFCTUB serves as a valuable resource for the research community, facilitating rigorous evaluation of MLLMs in the domain of CTU.