Multi-Level Aware Preference Learning: Enhancing RLHF for Complex Multi-Instruction Tasks

TL;DR

This paper introduces MAPL, a novel framework that leverages multi-level preference signals, including intra- and inter-sample preferences, to improve RLHF for complex multi-instruction tasks, reducing resource needs and bias.

Contribution

The paper proposes a new multi-level preference learning framework that captures latent signals within prompts and among samples, enhancing RLHF's effectiveness for complex instructions.

Findings

Improved performance on multiple benchmarks.

Enhanced multi-instruction task handling.

Effective integration with reward modeling and preference optimization.

Abstract

RLHF has emerged as a predominant approach for aligning artificial intelligence systems with human preferences, demonstrating exceptional and measurable efficacy in instruction following tasks; however, it exhibits insufficient compliance capabilities when confronted with complex multi-instruction tasks. Conventional approaches rely heavily on human annotation or more sophisticated large language models, thereby introducing substantial resource expenditure or potential bias concerns. Meanwhile, alternative synthetic methods that augment standard preference datasets often compromise the model's semantic quality. Our research identifies a critical oversight in existing techniques, which predominantly focus on comparing responses while neglecting valuable latent signals embedded within prompt inputs, and which only focus on preference disparities at the intra-sample level, while neglecting to account for the inter-sample level preference differentials that exist among preference data. To leverage these previously neglected indicators, we propose a novel Multi-level Aware Preference Learning (MAPL) framework, capable of enhancing multi-instruction capabilities. Specifically, for any given response in original preference data pairs, we construct varied prompts with a preference relation under different conditions, in order to learn intra-sample level preference disparities. Furthermore, for any given original preference pair, we synthesize multi-instruction preference pairs to capture preference discrepancies at the inter-sample level. Building on the two datasets constructed above, we consequently devise two sophisticated training objective functions. Subsequently, our framework integrates seamlessly into both Reward Modeling and Direct Preference Optimization paradigms. Through rigorous evaluation across multiple benchmarks, we empirically validate the efficacy of our framework.