Syllables to Scenes: Literary-Guided Free-Viewpoint 3D Scene Synthesis from Japanese Haiku

TL;DR

This paper presents HaikuVerse, a novel framework that translates Japanese Haiku poetry into detailed 3D scenes, combining literary analysis with advanced generative techniques to preserve emotional and visual fidelity in immersive environments.

Contribution

It introduces a hierarchical literary-criticism grounded parsing method and a multi-stage synthesis pipeline for accurate poetic-to-3D scene generation, advancing beyond existing text-to-3D approaches.

Findings

Outperforms traditional methods in literary fidelity

Produces higher quality and more coherent 3D scenes

Successfully captures emotional and visual nuances of poetry

Abstract

In the era of the metaverse, where immersive technologies redefine human experiences, translating abstract literary concepts into navigable 3D environments presents a fundamental challenge in preserving semantic and emotional fidelity. This research introduces HaikuVerse, a novel framework for transforming poetic abstraction into spatial representation, with Japanese Haiku serving as an ideal test case due to its sophisticated encapsulation of profound emotions and imagery within minimal text. While existing text-to-3D methods struggle with nuanced interpretations, we present a literary-guided approach that synergizes traditional poetry analysis with advanced generative technologies. Our framework centers on two key innovations: (1) Hierarchical Literary-Criticism Theory Grounded Parsing (H-LCTGP), which captures both explicit imagery and implicit emotional resonance through structured semantic decomposition, and (2) Progressive Dimensional Synthesis (PDS), a multi-stage pipeline that systematically transforms poetic elements into coherent 3D scenes through sequential diffusion processes, geometric optimization, and real-time enhancement. Extensive experiments demonstrate that HaikuVerse significantly outperforms conventional text-to-3D approaches in both literary fidelity and visual quality, establishing a new paradigm for preserving cultural heritage in immersive digital spaces. Project website at: https://syllables-to-scenes.github.io/