Decompose and Leverage Preferences from Expert Models for Improving Trustworthiness of MLLMs

TL;DR

This paper introduces DecompGen, a decomposable framework using open-source expert models to improve the trustworthiness of multimodal large language models by fine-grained preference assessment.

Contribution

DecompGen is a novel framework that decomposes responses into atomic tasks assessed by specialized experts, enabling better preference data construction and MLLM alignment.

Findings

DecompGen improves MLLMs' trustworthiness.

DGPref dataset enhances preference learning.

Open-source experts outperform closed models in evaluation.

Abstract

Multimodal Large Language Models (MLLMs) can enhance trustworthiness by aligning with human preferences. As human preference labeling is laborious, recent works employ evaluation models for assessing MLLMs' responses, using the model-based assessments to automate preference dataset construction. This approach, however, faces challenges with MLLMs' lengthy and compositional responses, which often require diverse reasoning skills that a single evaluation model may not fully possess. Additionally, most existing methods rely on closed-source models as evaluators. To address limitations, we propose DecompGen, a decomposable framework that uses an ensemble of open-sourced expert models. DecompGen breaks down each response into atomic verification tasks, assigning each task to an appropriate expert model to generate fine-grained assessments. The DecompGen feedback is used to automatically construct our preference dataset, DGPref. MLLMs aligned with DGPref via preference learning show improvements in trustworthiness, demonstrating the effectiveness of DecompGen.