Punny Captions: Witty Wordplay in Image Descriptions

TL;DR

This paper explores computational methods to generate witty, pun-based image descriptions, employing retrieval and neural network generation, and demonstrates that models can produce descriptions perceived as slightly wittier than human ones under constrained conditions.

Contribution

It introduces two novel approaches for generating witty image descriptions using puns, and provides human evaluation showing the models' effectiveness compared to baselines.

Findings

Models produce wittier descriptions than baseline methods.

Humans are generally wittier than models when unrestricted.

Model-generated descriptions are slightly wittier under constrained conditions.

Abstract

Wit is a form of rich interaction that is often grounded in a specific situation (e.g., a comment in response to an event). In this work, we attempt to build computational models that can produce witty descriptions for a given image. Inspired by a cognitive account of humor appreciation, we employ linguistic wordplay, specifically puns, in image descriptions. We develop two approaches which involve retrieving witty descriptions for a given image from a large corpus of sentences, or generating them via an encoder-decoder neural network architecture. We compare our approach against meaningful baseline approaches via human studies and show substantial improvements. We find that when a human is subject to similar constraints as the model regarding word usage and style, people vote the image descriptions generated by our model to be slightly wittier than human-written witty descriptions. Unsurprisingly, humans are almost always wittier than the model when they are free to choose the vocabulary, style, etc.

Figures

Full text

1 Introduction

“Wit is the sudden marriage of ideas which before their union were not perceived to have any relation.”

– Mark Twain

Witty remarks are often contextual, i.e., grounded in a specific situation. Developing computational models that can emulate rich forms of interaction like contextual humor, is a crucial step towards making human-AI interaction more natural and more engaging yu2016wizard. E.g., witty chatbots could help relieve stress and increase user engagement by being more personable and human-like. Bots could automatically post witty comments (or suggest witty responses) on social media, chat, or messaging.

The absence of large scale corpora of witty captions and the prohibitive cost of collecting such a dataset (being witty is harder than just describing an image) makes the problem of producing contextually witty image descriptions challenging.

In this work, we attempt to tackle the challenging task of producing witty (pun-based) remarks for a given (possibly boring) image. Our approach is inspired by a two-stage cognitive account of humor appreciation TwoStageModelHumor which states that a perceiver experiences humor when a stimulus such as a joke, captioned cartoon, etc., causes an incongruity, which is shortly followed by resolution.

We introduce an incongruity in the perceiver’s mind while describing an image by using an unexpected word that is phonetically similar (pun) to a concept related to the image. E.g., in Fig. 1(b), the expectations of a perceiver regarding the image (bear, stones, etc.) is momentarily disconfirmed by the (phonetically similar) word ‘bare’. This incongruity is resolved when the perceiver parses the entire image description. The incongruity followed by resolution can be perceived to be witty.111Indeed, a perceiver may fail to appreciate wit if the process of ‘solving’ (resolution) is trivial (the joke is obvious) or too complex (they do not ‘get’ the joke).

We build two computational models based on this approach to produce witty descriptions for an image. First, a model that retrieves sentences containing a pun that are relevant to the image from a large corpus of stories zhu2015aligning. Second, a model that generates witty descriptions for an image using a modified inference procedure during image captioning which includes the specified pun word in the description.

Our paper makes the following contributions: To the best of our knowledge, this is the first work that tackles the challenging problem of producing a witty natural language remark in an everyday (boring) context. We present two novel models to produce witty (pun-based) captions for a novel (likely boring) image. Our models rely on linguistic wordplay. They use an unexpected pun in an image description during inference/retrieval. Thus, they do not require to be trained with witty captions. Humans vote the descriptions from the top-ranked generated captions ‘wittier’ than three baseline approaches. Moreover, in a Turing test-style evaluation, our model’s best image description is found to be wittier than a witty human-written caption222 This data is available on the author’s webpage. 55% of the time when the human is subject to the same constraints as the machine regarding word usage and style.

2 Related Work

Humor theory. General Theory of Verbal Humor attardo1991script characterizes linguistic stimuli that induce humor but implementing computational models of it requires severely restricting its assumptions binsted1996machine.

Puns. zwicky1986imperfect classify puns as perfect (pronounced exactly the same) or imperfect (pronounced differently). Similarly, pepicello1984language categorize riddles based on the linguistic ambiguity that they exploit – phonological, morphological or syntactic.  Kao2013-kx formalize the notion of incongruity in puns and use a probabilistic model to evaluate the funniness of a sentence.  Jaech2016PhonologicalP learn phone-edit distances to predict the counterpart, given a pun by drawing from automatic speech recognition techniques. In contrast, we augment a web-scraped list of puns using an existing model of pronunciation similarity.

Generating textual humor. JAPE JAPE also uses phonological ambiguity to generate pun-based riddles. While our task involves producing free-form responses to a novel stimulus, JAPE produces stand-alone “canned” jokes. HAHAcronym HAHAcronym generates a funny expansion of a given acronym. Unlike our work, HAHAcronym operates on text, and is limited to producing sets of words.  UnsupervisedJokeBigData develop an unsupervised model that produces jokes of the form, “I like my X like I like my Y, Z” .

Generating multi-modal humor. meme predict a meme’s text based on a given funny image. Similarly, shahaf2015inside and radev2015humor learn to rank cartoon captions based on their funniness. Unlike typical, boring images in our task, memes and cartoons are images that are already funny or atypical. E.g., “LOL-cats” (funny cat photos), “Bieber-memes” (modified pictures of Justin Bieber), cartoons with talking animals, etc.  chandrasekaran2015we alter an abstract scene to make it more funny. In comparison, our task is to generate witty natural language remarks for a novel image.

Poetry generation. Although our tasks are different, our generation approach is conceptually similar to Ghazvininejad2016GeneratingTP who produce poetry, given a topic. While they also generate and score a set of candidates, their approach involves many more constraints and utilizes a finite state acceptor unlike our approach which enforces constraints during beam search of the RNN decoder.

3 Approach

Extracting tags. The first step in producing a contextually witty remark is to identify concepts that are relevant to the context (image). At times, these concepts are directly available as e.g., tags posted on social media. We consider the general case where such tags are unavailable, and automatically extract tags associated with an image.

We extract the top-5 object categories predicted by a state-of-the-art Inception-ResNet-v2 model szegedy2017inception trained for image classification on ImageNet deng2009imagenet. We also consider the words from a (boring) image description (generated from vinyals2016show). We combine the classifier object labels and words from the caption (ignoring stopwords) to produce a set of tags associated with an image, as shown in Fig. 2. We then identify concepts from this collection that can potentially induce wit.

Identifying puns. We attempt to induce an incongruity by using a pun in the image description. We identify candidate words for linguistic wordplay by comparing image tags against a list of puns.

We construct the list of puns by mining the web for differently spelled words that sound exactly the same (heterographic homophones). We increase coverage by also considering pairs of words with 0 edit-distance, according to a metric based on fine-grained articulatory representations (AR) of word pronunciations jyothi2014revisiting. Our list of puns has a total of 1067 unique words (931 from the web and 136 from the AR-based model).

The pun list yields a set of puns that are associated with a given image and their phonologically identical counterparts, which together form the pun vocabulary for the image. We evaluate our approach on the subset of images that have non-empty pun vocabularies (about 2 in 5 images).

Generating punny image captions. We introduce an incongruity by forcing a vanilla image captioning model vinyals2016show to decode a phonological counterpart of a pun word associated with the image, at a specific time-step during inference (e.g., ‘sell’ or ‘sighed’, showed in orange in Fig. 2). We achieve this by limiting the vocabulary of the decoder at that time-step to only contain counterparts of image-puns. In following time-steps, the decoder generates new words conditioned on all previously decoded words. Thus, the decoder attempts to generate sentences that flow well based on previously uttered words.

We train two models that decode an image description in forward (start to end) and reverse (end to start) directions, depicted as ‘fRNN’ and ‘rRNN’ in Fig. 2 respectively. The fRNN can decode words after accounting for the incongruity that occurs early in the sentence and the rRNN is able to decode the early words in the sentence after accounting for the incongruity that can occur later. The forward RNN and reverse RNN generate sentences in which the pun appears in each of the first $T$ and last $T$ positions, respectively.333For an image, we choose $T=\{1,2,...,5\}$ and beam size $=6$ for each decoder. This generates a pool of $5$ (T) $*$ $6$ (beam size) $*$ $2$ (forward + reverse decoder) $=60$ candidates.

Retrieving punny image captions. As an alternative to our approach of generating witty remarks for the given image, we also attempt to leverage natural, human-written sentences which are relevant (yet unexpected) in the given context. Concretely, we retrieve natural language sentences444To prevent the context of the sentence from distracting the perceiver, we consider sentences with $<15$ words. Overall, we are left with a corpus of about 13.5 million sentences. from a combination of the Book Corpus zhu2015aligning and corpora from the NLTK toolkit Loper2002. The retrieved sentences each (a) contains an incongruity (pun) whose counterpart is associated with the image, and (b) has support in the image (contains an image tag). This yields a pool of candidate captions that are perfectly grammatical, a little unexpected, and somewhat relevant to the image (see Sec. D).

Ranking. We rank captions in the candidate pools from both generation and retrieval models, according to their log-probability score under the image captioning model. We observe that the higher-ranked descriptions are more relevant to the image and grammatically correct. We then perform non-maximal suppression, i.e., eliminate captions that are similar555Two sentences are similar if the cosine similarity between the average of the Word2Vec word2vec representations of words in each sentence is $\geq$ 0.8. to a higher-ranked caption to reduce the pool to a smaller, more diverse set. We report results on the 3 top-ranked captions. We describe the effect of design choices in the supplementary.

4 Results

Data. We evaluate witty captions from our approach via human studies. 100 random images (having associated puns) are sampled from the validation set of COCO lin2014microsoft.

Baselines.

We compare the wittiness of descriptions generated by our model against 3 qualitatively different baselines, and a human-written witty description of an image. Each of these evaluates a different component of our approach. Regular inference generates a fluent caption that is relevant to the image but is not attempting to be witty. Witty mismatch is a human-written witty caption, but for a different image from the one being evaluated. This baseline results in a caption that is intended to be witty, but does not attempt to be relevant to the image. Ambiguous is a ‘punny’ caption where a pun word in the boring (regular) caption is replaced by its counterpart. This caption is likely to contain content that is relevant to the image, and it contains a pun. However, the pun is not being used in a fluent manner.

We evaluate the image-relevance of the top witty caption by comparing against a boring machine caption and a random caption (see supplementary).

Evaluating annotations. Our task is to generate captions that a layperson might find witty. To evaluate performance on this task, we ask people on Amazon Mechanical Turk (AMT) to vote for the wittier among the given pair of captions for an image. We collect annotations from 9 unique workers for each relative choice and take the majority vote as ground-truth. For each image, we compare each of the generated 3 top-ranked and 1 low-ranked caption against 3 baseline captions and 1 human-written witty caption.666This results in a total of $4$ (captions) $*2$ (generation $+$ retrieval) $*4$ (baselines $+$ human caption) $=32$ comparisons of our approach against baselines. We also compare the wittiness of the 4 generated captions against the 4 retrieved captions (see supplementary) for an image (16 comparisons). In total, we perform 48 comparisons per image, for 100 images.

Constrained human-written witty captions. We evaluate the ability of humans and automatic methods to use the given context and pun words to produce a caption that is perceived as witty. We ask subjects on AMT to describe a given image in a witty manner. To prevent observable structural differences between machine and human-written captions, we ensure consistent pun vocabulary (utilization of pre-specified puns for a given image). We also ask people to avoid first person accounts or quote characters in the image.

Metric.  3, we report performance of the generation approach using the Recall@K metric. For $K=1,2,3$, we plot the percentage of images for which at least one of the $K$ ‘best’ descriptions from our model outperformed another approach.

Generated captions vs. baselines. As we see in Fig. 3, the top generated image description (top-1G) is perceived as wittier compared to all baseline approaches more often than not (the vote is >$$50\% at $K=1$). We observe that as $K$ increases, the recall steadily increases, i.e., when we consider the top $K$ generated captions, increasingly often, humans find at least one of them to be wittier than captions produced by baseline approaches. People find the top-1G for a given image to be wittier than mismatched human-written image captions, about 95% of the time. The top-1G is also wittier than a naive approach that introduces ambiguity about 54.2% of the time. When compared to a typical, boring caption, the generated captions are wittier 68% of the time. Further, in a head-to-head comparison, the generated captions are wittier than the retrieved captions 67.7% of the time. We also validate our choice of ranking captions based on the image captioning model score. We observe that a ‘bad’ caption, i.e., one ranked lower by our model, is significantly less witty than the top 3 output captions.

Surprisingly, when the human is constrained to use the same words and style as the model, the generated descriptions from the model are found to be wittier for 55% of the images. Note that in a Turing test, a machine would equal human performance at 50%777Recall that this compares how a witty description is constructed, given the image and specific pun words. A Turing test-style evaluation that compares the overall wittiness of a machine and a human would refrain from constraining the human in any way.. This led us to speculate if the constraints placed on language and style might be restricting people’s ability to be witty. We confirmed this by evaluating free-form human captions.

Free-form Human-written Witty Captions.

We ask people on AMT to describe an image (using any vocabulary) in a manner that would be perceived as funny. As expected, when compared against automatic captions from our approach, human evaluators find free-form human captions to be wittier about 90% of the time compared to 45% in the case of constrained human witty captions. Clearly, human-level creative language with unconstrained sentence length, style, choice of puns, etc., makes a significant difference in the wittiness of a description. In contrast, our automatic approach is constrained by caption-like language, length, and a word-based pun list. Training models to intelligently navigate this creative freedom is an exciting open challenge.

Qualitative analysis.

The generated witty captions exhibit interesting features like alliteration (‘a bare black bear …’) in Fig. 1(b) and 6(c). At times, both the original pun (pole) and its counterpart (poll) make sense for the image (Fig. 1(a)). Occasionally, a pun is naively replaced by its counterpart (Fig. 6(b)) or rare puns are used (Fig. 6(d)). On the other hand, some descriptions (Fig. 4(e) and 4(h)) that are forced to utilize puns do not make sense. See supplementary for analysis of retrieval model.

5 Conclusion

We presented novel computational models inspired by cognitive accounts to address the challenging task of producing contextually witty descriptions for a given image. We evaluate the models via human-studies, in which they significantly outperform meaningful baseline approaches.

Acknowledgements

We thank Shubham Toshniwal for his advice regarding the automatic speech recognition model. This work was supported in part by: a NSF CAREER award, ONR YIP award, ONR Grant N00014-14-12713, PGA Family Foundation award, Google FRA, Amazon ARA, DARPA XAI grant to DP and NVIDIA GPU donations, Google FRA, IBM Faculty Award, and Bloomberg Data Science Research Grant to MB.