ContextIQ: A Multimodal Expert-Based Video Retrieval System for Contextual Advertising

TL;DR

ContextIQ is a multimodal video retrieval system tailored for contextual advertising, leveraging expert-based features across multiple modalities to improve accuracy without extensive joint training, thereby enhancing ad placement and safety.

Contribution

The paper introduces ContextIQ, a novel multimodal expert-based video retrieval system that outperforms or matches state-of-the-art models without joint training, specifically designed for the ad ecosystem.

Findings

Achieves better or comparable results to state-of-the-art models on retrieval benchmarks.

Utilizes multiple modalities for improved retrieval accuracy.

Supports brand safety and content filtering in advertising applications.

Abstract

Contextual advertising serves ads that are aligned to the content that the user is viewing. The rapid growth of video content on social platforms and streaming services, along with privacy concerns, has increased the need for contextual advertising. Placing the right ad in the right context creates a seamless and pleasant ad viewing experience, resulting in higher audience engagement and, ultimately, better ad monetization. From a technology standpoint, effective contextual advertising requires a video retrieval system capable of understanding complex video content at a very granular level. Current text-to-video retrieval models based on joint multimodal training demand large datasets and computational resources, limiting their practicality and lacking the key functionalities required for ad ecosystem integration. We introduce ContextIQ, a multimodal expert-based video retrieval system designed specifically for contextual advertising. ContextIQ utilizes modality-specific experts-video, audio, transcript (captions), and metadata such as objects, actions, emotion, etc.-to create semantically rich video representations. We show that our system, without joint training, achieves better or comparable results to state-of-the-art models and commercial solutions on multiple text-to-video retrieval benchmarks. Our ablation studies highlight the benefits of leveraging multiple modalities for enhanced video retrieval accuracy instead of using a vision-language model alone. Furthermore, we show how video retrieval systems such as ContextIQ can be used for contextual advertising in an ad ecosystem while also addressing concerns related to brand safety and filtering inappropriate content.