Evaluating Multimodal Large Language Models on Vertically Written Japanese Text

TL;DR

This paper evaluates the ability of multimodal large language models to understand vertically written Japanese text, revealing limitations and improvements through synthetic dataset training.

Contribution

It introduces a synthetic Japanese OCR dataset for vertical writing and demonstrates its effectiveness in enhancing MLLMs' vertical text understanding capabilities.

Findings

Existing MLLMs perform poorly on vertical Japanese text.

Training on the synthetic dataset improves model performance on vertical writing.

The datasets and code are publicly available for further research.

Abstract

Multimodal Large Language Models (MLLMs) have seen rapid advances in recent years and are now being applied to visual document understanding tasks. They are expected to process a wide range of document images across languages, including Japanese. Understanding documents from images requires models to read what are written in them. Since some Japanese documents are written vertically, support for vertical writing is essential. However, research specifically focused on vertically written Japanese text remains limited. In this study, we evaluate the reading capability of existing MLLMs on vertically written Japanese text. First, we generate a synthetic Japanese OCR dataset by rendering Japanese texts into images, and use it for both model fine-tuning and evaluation. This dataset includes Japanese text in both horizontal and vertical writing. We also create an evaluation dataset sourced from the real-world document images containing vertically written Japanese text. Using these datasets, we demonstrate that the existing MLLMs perform worse on vertically written Japanese text than on horizontally written Japanese text. Furthermore, we show that training MLLMs on our synthesized Japanese OCR dataset results in improving the performance of models that previously could not handle vertical writing. The datasets and code are publicly available https://github.com/llm-jp/eval_vertical_ja.