Semi-supervised Chinese Poem-to-Painting Generation via Cycle-consistent Adversarial Networks

TL;DR

This paper introduces a semi-supervised cycle-consistent adversarial network approach for Chinese poem-to-painting generation, effectively leveraging limited paired data and large unpaired datasets to capture artistic symbolism.

Contribution

It proposes a novel semi-supervised model with cycle consistency for Chinese poem-to-painting translation, addressing data scarcity and improving semantic alignment.

Findings

Outperforms previous methods on CPDD dataset

Effectively captures symbolic artistic expression

Introduces new evaluation metrics for quality and diversity

Abstract

Classical Chinese poetry and painting represent the epitome of artistic expression, but the abstract and symbolic nature of their relationship poses a significant challenge for computational translation. Most existing methods rely on large-scale paired datasets, which are scarce in this domain. In this work, we propose a semi-supervised approach using cycle-consistent adversarial networks to leverage the limited paired data and large unpaired corpus of poems and paintings. The key insight is to learn bidirectional mappings that enforce semantic alignment between the visual and textual modalities. We introduce novel evaluation metrics to assess the quality, diversity, and consistency of the generated poems and paintings. Extensive experiments are conducted on a new Chinese Painting Description Dataset (CPDD). The proposed model outperforms previous methods, showing promise in capturing the symbolic essence of artistic expression. Codes are available online \url{https://github.com/Mnster00/poemtopainting}.