Caption Generation for Dongba Paintings via Prompt Learning and Semantic Fusion

TL;DR

This paper introduces PVGF-DPC, a novel framework for generating culturally accurate captions for Dongba paintings by integrating semantic prompts and fusion loss, addressing domain shift issues in traditional captioning models.

Contribution

It proposes a new encoder-decoder model with prompt learning and a fusion loss to improve captioning of culturally specific Dongba paintings, supported by a dedicated dataset.

Findings

Enhanced caption accuracy for Dongba paintings.

Effective integration of cultural cues into caption generation.

Improved semantic alignment between images and descriptions.

Abstract

Dongba paintings, the treasured pictorial legacy of the Naxi people in southwestern China, feature richly layered visual elements, vivid color palettes, and pronounced ethnic and regional cultural symbolism, yet their automatic textual description remains largely unexplored owing to severe domain shift when mainstream captioning models are applied directly. This paper proposes \textbf{PVGF-DPC} (\textit{Prompt and Visual Semantic-Generation Fusion-based Dongba Painting Captioning}), an encoder-decoder framework that integrates a content prompt module with a novel visual semantic-generation fusion loss to bridge the gap between generic natural-image captioning and the culturally specific imagery found in Dongba art. A MobileNetV2 encoder extracts discriminative visual features, which are injected into the layer normalization of a 10-layer Transformer decoder initialized with pretrained BERT weights; meanwhile, the content prompt module maps the image feature vector to culture-aware labels -- such as \emph{deity}, \emph{ritual pattern}, or \emph{hell ghost} -- and constructs a post-prompt that steers the decoder toward thematically accurate descriptions. The visual semantic-generation fusion loss jointly optimizes the cross-entropy objectives of both the prompt predictor and the caption generator, encouraging the model to extract key cultural and visual cues and to produce captions that are semantically aligned with the input image. We construct a dedicated Dongba painting captioning dataset comprising 9{}408 augmented images with culturally grounded annotations spanning seven thematic categories.