Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Models

TL;DR

FiFA is an automated data filtering method that improves the efficiency and effectiveness of fine-tuning text-to-image diffusion models with human feedback, reducing data requirements and enhancing model alignment.

Contribution

The paper introduces FiFA, a novel automated filtering algorithm that optimizes data selection for human feedback-based model fine-tuning, incorporating preference margin, text quality, and diversity.

Findings

Significantly improves training stability and performance.

Achieves 17% human preference with less than 0.5% data.

Reduces GPU hours to 1% of full dataset training.

Abstract

Fine-tuning text-to-image diffusion models with human feedback is an effective method for aligning model behavior with human intentions. However, this alignment process often suffers from slow convergence due to the large size and noise present in human feedback datasets. In this work, we propose FiFA, a novel automated data filtering algorithm designed to enhance the fine-tuning of diffusion models using human feedback datasets with direct preference optimization (DPO). Specifically, our approach selects data by solving an optimization problem to maximize three components: preference margin, text quality, and text diversity. The concept of preference margin is used to identify samples that are highly informative in addressing the noisy nature of feedback dataset, which is calculated using a proxy reward model. Additionally, we incorporate text quality, assessed by large language models to prevent harmful contents, and consider text diversity through a k-nearest neighbor entropy estimator to improve generalization. Finally, we integrate all these components into an optimization process, with approximating the solution by assigning importance score to each data pair and selecting the most important ones. As a result, our method efficiently filters data automatically, without the need for manual intervention, and can be applied to any large-scale dataset. Experimental results show that FiFA significantly enhances training stability and achieves better performance, being preferred by humans 17% more, while using less than 0.5% of the full data and thus 1% of the GPU hours compared to utilizing full human feedback datasets.