Multi-domain Multi-modal Document Classification Benchmark with a Multi-level Taxonomy

TL;DR

This paper introduces MMM-Bench, a comprehensive benchmark for multi-level, multi-domain, multi-modal document classification, featuring hierarchical taxonomy and real-world data from Alibaba to advance industrial document understanding.

Contribution

The paper presents the first benchmark with a hierarchical taxonomy and multi-domain data, addressing limitations of existing simplified document classification benchmarks.

Findings

Established baseline models and API-based approaches on MMM-Bench.

Identified four fundamental challenges in multi-level, multi-domain document classification.

Provided insights to guide future research in complex document understanding.

Abstract

Document classification forms the backbone of modern enterprise content management, yet existing benchmarks remain trapped in oversimplified paradigms -- single domain settings with flat label structures -- that bear little resemblance to the hierarchical, multi-modal, and cross-domain nature of real-world business documents. This gap not only misrepresents practical complexity but also stifles progress toward industrially viable document intelligence. To bridge this gap, we construct the first Multi-level, Multi-domain, Multi-modal document classification Benchmark (MMM-Bench). MMM-Bench includes (1) a deeply hierarchical taxonomy spanning five levels that capture the authentic organizational logic of business documentation; and (2) 5,990 real-world multi-modal documents meticulously curated from 12 commercial domains in Alibaba. Each document is manually annotated with a complete hierarchical path by domain experts. We establish comprehensive baselines on MMM-Bench, which consists of open-weight models and API-based models. Through systematic experiments, we identify four fundamental challenges within MMM-Bench and propose corresponding insights. To provide a solid foundation for advancing research in multi-level, multi-domain document classification, we release all of the data and the evaluation toolkit of MMM-Bench at https://github.com/MMMDC-Bench/MMMDC-Bench.