DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception

TL;DR

DEEM enhances large multimodal models' image perception by leveraging diffusion models' generative feedback, improving robustness against out-of-distribution data and reducing hallucinations without additional training modules.

Contribution

The paper introduces DEEM, a novel approach that uses diffusion models to improve the semantic alignment of image encoders in large multimodal models, addressing out-of-distribution challenges.

Findings

DEEM achieves up to 12.8% improvement on the POPE benchmark.

DEEM reduces visual hallucinations and enhances perception robustness.

Fewer training parameters and less data are needed compared to state-of-the-art methods.

Abstract

The development of large language models (LLMs) has significantly advanced the emergence of large multimodal models (LMMs). While LMMs have achieved tremendous success by promoting the synergy between multimodal comprehension and creation, they often face challenges when confronted with out-of-distribution data, such as which can hardly distinguish orientation, quantity, color, structure, etc. This is primarily due to their reliance on image encoders trained to encode images into task-relevant features, which may lead them to disregard irrelevant details. Delving into the modeling capabilities of diffusion models for images naturally prompts the question: Can diffusion models serve as the eyes of large language models for image perception? In this paper, we propose DEEM, a simple but effective approach that utilizes the generative feedback of diffusion models to align the semantic distributions of the image encoder. This addresses the drawbacks of previous methods that solely relied on image encoders like CLIP-ViT, thereby enhancing the model's resilience against out-of-distribution samples and reducing visual hallucinations. Importantly, this is achieved without requiring additional training modules and with fewer training parameters. We extensively evaluated DEEM on both our newly constructed RobustVQA benchmark and other well-known benchmarks, POPE and MMVP, for visual hallucination and perception. In particular, DEEM improves LMM's visual perception performance to a large extent (e.g., 4% higher on RobustVQA, 6.5% higher on MMVP and 12.8 % higher on POPE ). Compared to the state-of-the-art interleaved content generation models, DEEM exhibits enhanced robustness and a superior capacity to alleviate model hallucinations while utilizing fewer trainable parameters, less pre-training data (10%), and a smaller base model size.