Poetry in Pixels: Prompt Tuning for Poem Image Generation via Diffusion Models

TL;DR

This paper introduces PoemToPixel, a novel framework that uses prompt tuning and a new algorithm to generate images that visually interpret poems, supported by a new dataset of children's poems and images.

Contribution

It presents the PoemToPixel framework with the PoeKey algorithm for extracting key poetic elements to improve image generation from poetry.

Findings

Effective alignment of generated images with poetic content

Introduction of MiniPo dataset for diverse poetry-image pairs

Quantitative and qualitative validation of the framework

Abstract

The task of text-to-image generation has encountered significant challenges when applied to literary works, especially poetry. Poems are a distinct form of literature, with meanings that frequently transcend beyond the literal words. To address this shortcoming, we propose a PoemToPixel framework designed to generate images that visually represent the inherent meanings of poems. Our approach incorporates the concept of prompt tuning in our image generation framework to ensure that the resulting images closely align with the poetic content. In addition, we propose the PoeKey algorithm, which extracts three key elements in the form of emotions, visual elements, and themes from poems to form instructions which are subsequently provided to a diffusion model for generating corresponding images. Furthermore, to expand the diversity of the poetry dataset across different genres and ages, we introduce MiniPo, a novel multimodal dataset comprising 1001 children's poems and images. Leveraging this dataset alongside PoemSum, we conducted both quantitative and qualitative evaluations of image generation using our PoemToPixel framework. This paper demonstrates the effectiveness of our approach and offers a fresh perspective on generating images from literary sources.