DEJIMA: A Novel Large-scale Japanese Dataset for Image Captioning and Visual Question Answering

TL;DR

This paper introduces DEJIMA, a large-scale Japanese dataset for image captioning and VQA, created through a scalable pipeline, significantly surpassing existing datasets in size and cultural relevance, and improving model performance.

Contribution

The paper presents a novel scalable pipeline for constructing large-scale Japanese V&L datasets, resulting in DEJIMA, which enhances linguistic naturalness and cultural coverage compared to previous datasets.

Findings

DEJIMA contains 3.88 million image-text pairs.

Models trained on DEJIMA outperform baselines on Japanese V&L benchmarks.

DEJIMA demonstrates higher Japaneseness and cultural relevance than translated datasets.

Abstract

This work addresses the scarcity of high-quality, large-scale resources for Japanese Vision-and-Language (V&L) modeling. We present a scalable and reproducible pipeline that integrates large-scale web collection with rigorous filtering/deduplication, object-detection-driven evidence extraction, and Large Language Model (LLM)-based refinement under grounding constraints. Using this pipeline, we build two resources: an image-caption dataset (DEJIMA-Cap) and a VQA dataset (DEJIMA-VQA), each containing 3.88M image-text pairs, far exceeding the size of existing Japanese V&L datasets. Human evaluations demonstrate that DEJIMA achieves substantially higher Japaneseness and linguistic naturalness than datasets constructed via translation or manual annotation, while maintaining factual correctness at a level comparable to human-annotated corpora. Quantitative analyses of image feature distributions further confirm that DEJIMA broadly covers diverse visual domains characteristic of Japan, complementing its linguistic and cultural representativeness. Models trained on DEJIMA exhibit consistent improvements across multiple Japanese multimodal benchmarks, confirming that culturally grounded, large-scale resources play a key role in enhancing model performance. All data sources and modules in our pipeline are licensed for commercial use, and we publicly release the resulting dataset and metadata to encourage further research and industrial applications in Japanese V&L modeling.