Language Models Can See Better: Visual Contrastive Decoding For LLM Multimodal Reasoning

TL;DR

This paper introduces MVCD, a novel decoding framework that enhances large language models' visual perception and reasoning by converting visual signals into text and using contrastive decoding, without additional training.

Contribution

The paper proposes a modular visual contrastive decoding method that improves LLMs' multimodal reasoning without extra training, leveraging in-context learning and contrastive examples.

Findings

Consistent accuracy improvements across multiple datasets

Effective highlighting of new information through contrastive decoding

Enhanced visual reasoning capabilities in LLMs

Abstract

Although Large Language Models (LLMs) excel in reasoning and generation for language tasks, they are not specifically designed for multimodal challenges. Training Multimodal Large Language Models (MLLMs), however, is resource-intensive and constrained by various training limitations. In this paper, we propose the Modular-based Visual Contrastive Decoding (MVCD) framework to move this obstacle. Our framework leverages LLMs' In-Context Learning (ICL) capability and the proposed visual contrastive-example decoding (CED), specifically tailored for this framework, without requiring any additional training. By converting visual signals into text and focusing on contrastive output distributions during decoding, we can highlight the new information introduced by contextual examples, explore their connections, and avoid over-reliance on prior encoded knowledge. MVCD enhances LLMs' visual perception to make it see and reason over the input visuals. To demonstrate MVCD's effectiveness, we conduct experiments with four LLMs across five question answering datasets. Our results not only show consistent improvement in model accuracy but well explain the effective components inside our decoding strategy. Our code will be available at https://github.com/Pbhgit/MVCD.