Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution

TL;DR

This paper introduces a novel self-evolution framework for multimodal large language models that autonomously generates high-quality data from unannotated images, reducing reliance on human or GPT annotations and improving alignment.

Contribution

The proposed multimodal self-evolution method enables models to generate and evaluate data independently, enhancing alignment without external annotations or additional models.

Findings

Competitive performance with external-data methods

Efficient and scalable data generation process

Reduced hallucinations through content alignment

Abstract

Human preference alignment can greatly enhance Multimodal Large Language Models (MLLMs), but collecting high-quality preference data is costly. A promising solution is the self-evolution strategy, where models are iteratively trained on data they generate. However, current techniques still rely on human- or GPT-annotated data and sometimes require additional models or ground truth answers. To address these issues, we propose a novel multimodal self-evolution framework that enables the model to autonomously generate high-quality questions and answers using only unannotated images. First, we implement an image-driven self-questioning mechanism, allowing the model to create and evaluate questions based on image content, regenerating them if they are irrelevant or unanswerable. This sets a strong foundation for answer generation. Second, we introduce an answer self-enhancement technique, starting with image captioning to improve answer quality. We also use corrupted images to generate rejected answers, forming distinct preference pairs for optimization. Finally, we incorporate an image content alignment loss function alongside Direct Preference Optimization (DPO) loss to reduce hallucinations, ensuring the model focuses on image content. Experiments show that our framework performs competitively with methods using external information, offering a more efficient and scalable approach to MLLMs.