Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment

TL;DR

This paper introduces FiSAO, a self-alignment method that uses the model's visual encoder as a fine-grained verifier, improving vision-language alignment without extra data and reducing hallucinations.

Contribution

The paper presents FiSAO, a novel token-level self-alignment technique that enhances VLLMs' alignment by leveraging internal visual feedback, eliminating the need for external datasets.

Findings

FiSAO outperforms traditional preference tuning methods.

Token-level feedback significantly improves alignment quality.

FiSAO effectively reduces hallucinations in VLLMs.

Abstract

The recent advancements in large language models (LLMs) and pre-trained vision models have accelerated the development of vision-language large models (VLLMs), enhancing the interaction between visual and linguistic modalities. Despite their notable success across various domains, VLLMs face challenges in modality alignment, which can lead to issues like hallucinations and unsafe content generation. Current alignment techniques often rely on coarse feedback and external datasets, limiting scalability and performance. In this paper, we propose FiSAO (Fine-Grained Self-Alignment Optimization), a novel self-alignment method that utilizes the model's own visual encoder as a fine-grained verifier to improve vision-language alignment without the need for additional data. By leveraging token-level feedback from the vision encoder, FiSAO significantly improves vision-language alignment, even surpassing traditional preference tuning methods that require additional data. Through both theoretical analysis and experimental validation, we demonstrate that FiSAO effectively addresses the misalignment problem in VLLMs, marking the first instance of token-level rewards being applied to such models.