OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference

TL;DR

This paper introduces OmniAlign-V, a large dataset and benchmark aimed at improving multi-modal large language models' alignment with human preferences, demonstrating significant enhancements through finetuning methods.

Contribution

The paper presents OmniAlign-V, a new high-quality dataset and MM-AlignBench, a benchmark for aligning MLLMs with human values, along with experimental validation.

Findings

Finetuning with OmniAlign-V improves human preference alignment.

Enhances alignment without compromising standard VQA performance.

Provides publicly available datasets, benchmark, code, and checkpoints.

Abstract

Recent advancements in open-source multi-modal large language models (MLLMs) have primarily focused on enhancing foundational capabilities, leaving a significant gap in human preference alignment. This paper introduces OmniAlign-V, a comprehensive dataset of 200K high-quality training samples featuring diverse images, complex questions, and varied response formats to improve MLLMs' alignment with human preferences. We also present MM-AlignBench, a human-annotated benchmark specifically designed to evaluate MLLMs' alignment with human values. Experimental results show that finetuning MLLMs with OmniAlign-V, using Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO), significantly enhances human preference alignment while maintaining or enhancing performance on standard VQA benchmarks, preserving their fundamental capabilities. Our datasets, benchmark, code and checkpoints have been released at https://github.com/PhoenixZ810/OmniAlign-V.