Don't Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models

TL;DR

This paper critically evaluates contrastive vision-and-language models' ability to understand complex advertisements, revealing that current benchmarks can be fooled and emphasizing the need for more robust evaluation methods.

Contribution

The authors introduce TRADE, an adversarial dataset for evaluating multimodal reasoning in ad understanding, exposing limitations of existing models.

Findings

Contrastive VLMs exploit grounding heuristics to solve ad understanding tasks.

TRADE dataset reveals models are fooled by implausible explanations.

Current benchmarks do not adequately measure true multimodal reasoning.

Abstract

Image-based advertisements are complex multimodal stimuli that often contain unusual visual elements and figurative language. Previous research on automatic ad understanding has reported impressive zero-shot accuracy of contrastive vision-and-language models (VLMs) on an ad-explanation retrieval task. Here, we examine the original task setup and show that contrastive VLMs can solve it by exploiting grounding heuristics. To control for this confound, we introduce TRADE, a new evaluation test set with adversarial grounded explanations. While these explanations look implausible to humans, we show that they "fool" four different contrastive VLMs. Our findings highlight the need for an improved operationalisation of automatic ad understanding that truly evaluates VLMs' multimodal reasoning abilities. We make our code and TRADE available at https://github.com/dmg-illc/trade .