MM-AU:Towards Multimodal Understanding of Advertisement Videos

TL;DR

This paper introduces MM-AU, a comprehensive multimodal benchmark for understanding advertisement videos across topics, tone transitions, and social messages, demonstrating the effectiveness of multimodal transformers over unimodal methods.

Contribution

The paper presents MM-AU, a large multilingual multimodal benchmark for ad understanding, and shows that multimodal transformer models outperform unimodal baselines in key tasks.

Findings

Multimodal models outperform unimodal approaches in ad understanding.

Zero-shot reasoning with large language models provides baseline insights.

MM-AU includes over 8.4K videos for comprehensive evaluation.

Abstract

Advertisement videos (ads) play an integral part in the domain of Internet e-commerce as they amplify the reach of particular products to a broad audience or can serve as a medium to raise awareness about specific issues through concise narrative structures. The narrative structures of advertisements involve several elements like reasoning about the broad content (topic and the underlying message) and examining fine-grained details involving the transition of perceived tone due to the specific sequence of events and interaction among characters. In this work, to facilitate the understanding of advertisements along the three important dimensions of topic categorization, perceived tone transition, and social message detection, we introduce a multimodal multilingual benchmark called MM-AU composed of over 8.4K videos (147 hours) curated from multiple web sources. We explore multiple zero-shot reasoning baselines through the application of large language models on the ads transcripts. Further, we demonstrate that leveraging signals from multiple modalities, including audio, video, and text, in multimodal transformer-based supervised models leads to improved performance compared to unimodal approaches.