Analyzing Multimodal Objectives Through the Lens of Generative Diffusion Guidance

TL;DR

This paper investigates how different multimodal learning objectives like contrastive, matching, and captioning influence semantic understanding in generative diffusion models, providing insights and a simple baseline for improved guidance.

Contribution

It introduces a novel analysis framework comparing multimodal objectives through diffusion models and proposes a straightforward baseline that enhances generative guidance quality.

Findings

Contrastive, matching, and captioning objectives encode distinct semantic signals.

The proposed baseline improves the quality of generative guidance.

Analysis reveals how objectives influence semantic signal extraction.

Abstract

Recent years have witnessed astonishing advances in the field of multimodal representation learning, with contrastive learning being the cornerstone for major breakthroughs. Latest works delivered further improvements by incorporating different objectives such as masked modeling and captioning into the frameworks, but our understanding on how these objectives facilitate learning remains vastly incomplete. In this paper, we leverage the fact that classifier-guided diffusion models generate images that reflect the semantic signals provided by the classifier to study the characteristics of multimodal learning objectives. Specifically, we compare contrastive, matching and captioning loss in terms of their semantic signals, and introduce a simple baseline that not only supports our analyses but also improves the quality of generative guidance in a straightforward manner.