How to Describe Images in a More Funny Way? Towards a Modular Approach to Cross-Modal Sarcasm Generation

TL;DR

This paper introduces a novel modular approach for generating sarcastic descriptions of images without paired training data, combining extraction, generation, and ranking to produce high-quality, humorous cross-modal content.

Contribution

It proposes the Extraction-Generation-Ranking (EGRM) method for cross-modal sarcasm generation, addressing data scarcity and the need for imaginative, inconsistent image-text outputs.

Findings

Outperforms existing methods in human evaluations

Generates diverse and humorous sarcastic descriptions

Effective ranking improves sarcasm quality

Abstract

Sarcasm generation has been investigated in previous studies by considering it as a text-to-text generation problem, i.e., generating a sarcastic sentence for an input sentence. In this paper, we study a new problem of cross-modal sarcasm generation (CMSG), i.e., generating a sarcastic description for a given image. CMSG is challenging as models need to satisfy the characteristics of sarcasm, as well as the correlation between different modalities. In addition, there should be some inconsistency between the two modalities, which requires imagination. Moreover, high-quality training data is insufficient. To address these problems, we take a step toward generating sarcastic descriptions from images without paired training data and propose an Extraction-Generation-Ranking based Modular method (EGRM) for cross-model sarcasm generation. Specifically, EGRM first extracts diverse information from an image at different levels and uses the obtained image tags, sentimental descriptive caption, and commonsense-based consequence to generate candidate sarcastic texts. Then, a comprehensive ranking algorithm, which considers image-text relation, sarcasticness, and grammaticality, is proposed to select a final text from the candidate texts. Human evaluation at five criteria on a total of 1200 generated image-text pairs from eight systems and auxiliary automatic evaluation show the superiority of our method.