Crossing Borders: A Multimodal Challenge for Indian Poetry Translation and Image Generation

TL;DR

This paper introduces TAI, a framework combining translation and image generation to make Indian poetry accessible globally, using LLMs and diffusion models, and provides a new dataset for low-resource Indian languages.

Contribution

The paper presents a novel multimodal framework for translating and visually representing Indian poetry, including a new dataset for low-resource languages and a semantic graph-based image generation approach.

Findings

TAI Diffusion outperforms baselines in poem image generation

The Odds Ratio Preference Alignment Algorithm improves translation accuracy

The Morphologically Rich Indian Language Poems MorphoVerse Dataset supports low-resource languages

Abstract

Indian poetry, known for its linguistic complexity and deep cultural resonance, has a rich and varied heritage spanning thousands of years. However, its layered meanings, cultural allusions, and sophisticated grammatical constructions often pose challenges for comprehension, especially for non-native speakers or readers unfamiliar with its context and language. Despite its cultural significance, existing works on poetry have largely overlooked Indian language poems. In this paper, we propose the Translation and Image Generation (TAI) framework, leveraging Large Language Models (LLMs) and Latent Diffusion Models through appropriate prompt tuning. Our framework supports the United Nations Sustainable Development Goals of Quality Education (SDG 4) and Reduced Inequalities (SDG 10) by enhancing the accessibility of culturally rich Indian-language poetry to a global audience. It includes (1) a translation module that uses an Odds Ratio Preference Alignment Algorithm to accurately translate morphologically rich poetry into English, and (2) an image generation module that employs a semantic graph to capture tokens, dependencies, and semantic relationships between metaphors and their meanings, to create visually meaningful representations of Indian poems. Our comprehensive experimental evaluation, including both human and quantitative assessments, demonstrates the superiority of TAI Diffusion in poem image generation tasks, outperforming strong baselines. To further address the scarcity of resources for Indian-language poetry, we introduce the Morphologically Rich Indian Language Poems MorphoVerse Dataset, comprising 1,570 poems across 21 low-resource Indian languages. By addressing the gap in poetry translation and visual comprehension, this work aims to broaden accessibility and enrich the reader's experience.