Benchmarking VLMs' Reasoning About Persuasive Atypical Images

TL;DR

This paper benchmarks vision-language models' ability to understand persuasive atypical images in ads, revealing their limited reasoning skills and proposing methods for improved atypicality-aware descriptions.

Contribution

Introduces three novel tasks to evaluate VLMs' understanding of atypicality in persuasive images and explores atypicality-aware verbalization techniques.

Findings

VLMs lack advanced reasoning compared to LLMs

Simple strategies effectively extract atypicality information

Atypicality improves understanding of persuasive ads

Abstract

Vision language models (VLMs) have shown strong zero-shot generalization across various tasks, especially when integrated with large language models (LLMs). However, their ability to comprehend rhetorical and persuasive visual media, such as advertisements, remains understudied. Ads often employ atypical imagery, using surprising object juxtapositions to convey shared properties. For example, Fig. 1 (e) shows a beer with a feather-like texture. This requires advanced reasoning to deduce that this atypical representation signifies the beer's lightness. We introduce three novel tasks, Multi-label Atypicality Classification, Atypicality Statement Retrieval, and Aypical Object Recognition, to benchmark VLMs' understanding of atypicality in persuasive images. We evaluate how well VLMs use atypicality to infer an ad's message and test their reasoning abilities by employing semantically challenging negatives. Finally, we pioneer atypicality-aware verbalization by extracting comprehensive image descriptions sensitive to atypical elements. Our findings reveal that: (1) VLMs lack advanced reasoning capabilities compared to LLMs; (2) simple, effective strategies can extract atypicality-aware information, leading to comprehensive image verbalization; (3) atypicality aids persuasive advertisement understanding. Code and data will be made available.