Self-supervised Attribute-aware Dynamic Preference Ranking Alignment

TL;DR

This paper introduces SeAdpra, a self-supervised, attribute-aware method for dynamic preference ranking that improves list-level alignment in response generation without relying on costly human annotations.

Contribution

It proposes a novel self-supervised approach using Attribute-Perceptual Distance Factors for fine-grained preference learning and introduces scalable evaluation metrics and a challenging dataset.

Findings

SeAdpra outperforms existing methods on multiple datasets.

It achieves better alignment with human preferences.

The approach demonstrates strong generalizability across domains.

Abstract

Reinforcement Learning from Human Feedback and its variants excel in aligning with human intentions to generate helpful, harmless, and honest responses. However, most of them rely on costly human-annotated pairwise comparisons for supervised alignment, which is not suitable for list-level scenarios, such as community question answering. Additionally, human preferences are influenced by multiple intrinsic factors in responses, leading to decision-making inconsistencies. Therefore, we propose \textbf{Se}lf-supervised \textbf{A}ttribute-aware \textbf{d}ynamic \textbf{p}reference \textbf{ra}nking, called \shortname. \ It quantifies preference differences between responses based on Attribute-Perceptual Distance Factors (APDF) and dynamically determines the list-wise alignment order. Furthermore, it achieves fine-grained preference difference learning and enables precise alignment with the optimal one. We specifically constructed a challenging code preference dataset named StaCoCoQA, and introduced more cost-effective and scalable preference evaluation metrics: PrefHit and PrefRecall. Extensive experimental results show that SeAdpra exhibits superior performance and generalizability on both StaCoCoQA and preference datasets from eight popular domains.