VACoDe: Visual Augmented Contrastive Decoding

TL;DR

VACoDe is a novel method that adaptively selects the most contrasting image augmentation to improve the accuracy of vision-language models without extra training or external data.

Contribution

It introduces VACoDe, which utilizes multiple augmentations and a softmax distance metric to enhance contrastive decoding in vision-language models.

Findings

Outperforms previous contrastive decoding methods.

Improves output quality across various vision-language tasks.

Universal applicability without additional training or external data.

Abstract

Despite the astonishing performance of recent Large Vision-Language Models (LVLMs), these models often generate inaccurate responses. To address this issue, previous studies have focused on mitigating hallucinations by employing contrastive decoding (CD) with augmented images, which amplifies the contrast with the original image. However, these methods have limitations, including reliance on a single augmentation, which is restrictive for certain tasks, as well as the high cost of using external knowledge. In this study, we address these limitations by exploring how to utilize multiple image augmentations. Through extensive experiments, we observed that different augmentations produce varying levels of contrast depending on the task. Based on this observation, we introduce a novel method called VACoDe, Visual Augmented Contrastive Decoding. This method adaptively selects the augmentation with the highest contrast for each task using the proposed softmax distance metric. Our empirical tests show that \alg outperforms previous methods and improves output quality in various vision-language tasks. Additionally, VACoDe can be universally applied across different model types and sizes without additional training or the use of external models and data.