CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs

TL;DR

This paper introduces CLIP-DPO, a novel method leveraging contrastively pre-trained vision-language models like CLIP to reduce hallucinations in LVLMs without external data or APIs, improving robustness and grounding.

Contribution

The paper presents CLIP-DPO, a preference optimization approach that uses CLIP embeddings for fine-tuning LVLMs, effectively reducing hallucinations without additional training data or external models.

Findings

Significant hallucination reduction in MobileVLM-v2 and LlaVA-1.5 models.

Improved zero-shot classification performance.

Preservation of original benchmark performance.

Abstract

Despite recent successes, LVLMs or Large Vision Language Models are prone to hallucinating details like objects and their properties or relations, limiting their real-world deployment. To address this and improve their robustness, we present CLIP-DPO, a preference optimization method that leverages contrastively pre-trained Vision-Language (VL) embedding models, such as CLIP, for DPO-based optimization of LVLMs. Unlike prior works tackling LVLM hallucinations, our method does not rely on paid-for APIs, and does not require additional training data or the deployment of other external LVLMs. Instead, starting from the initial pool of supervised fine-tuning data, we generate a diverse set of predictions, which are ranked based on their CLIP image-text similarities, and then filtered using a robust rule-based approach to obtain a set of positive and negative pairs for DPO-based training. We applied CLIP-DPO fine-tuning to the MobileVLM-v2 family of models and to LlaVA-1.5, in all cases observing significant improvements in terms of hallucination reduction over baseline models. We also observe better performance for zero-shot classification, suggesting improved grounding capabilities, and verify that the original performance on standard LVLM benchmarks is overall preserved.