Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languages

TL;DR

This paper introduces Chitrakshara, a large multilingual multimodal dataset for Indian languages, aiming to improve vision-language models' representation and understanding of diverse Indian languages and multi-image reasoning.

Contribution

The paper presents the creation of the Chitrakshara dataset series, including large-scale multilingual image-text datasets for Indian languages, with detailed data collection, filtering, and analysis methods.

Findings

Chitrakshara dataset covers 11 Indian languages with extensive image-text pairs.

The dataset demonstrates high diversity and quality across Indic languages.

Potential to enhance culturally inclusive vision-language models.

Abstract

Multimodal research has predominantly focused on single-image reasoning, with limited exploration of multi-image scenarios. Recent models have sought to enhance multi-image understanding through large-scale pretraining on interleaved image-text datasets. However, most Vision-Language Models (VLMs) are trained primarily on English datasets, leading to inadequate representation of Indian languages. To address this gap, we introduce the Chitrakshara dataset series, covering 11 Indian languages sourced from Common Crawl. It comprises (1) Chitrakshara-IL, a large-scale interleaved pretraining dataset with 193M images, 30B text tokens, and 50M multilingual documents, and (2) Chitrakshara-Cap, which includes 44M image-text pairs with 733M tokens. This paper details the data collection pipeline, including curation, filtering, and processing methodologies. Additionally, we present a comprehensive quality and diversity analysis to assess the dataset's representativeness across Indic languages and its potential for developing more culturally inclusive VLMs.