Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equity

TL;DR

This paper introduces Fair-PP, a synthetic dataset based on social survey data, to improve alignment of large language models with personalized social equity preferences, using an automated data generation framework and reweighting methods.

Contribution

The paper presents a novel synthetic dataset and an automated framework for personalized preference data generation, addressing limitations of existing datasets in social equity alignment.

Findings

Fair-PP effectively captures diverse social equity preferences.

The reweighting method improves LLM alignment with target personas.

Empirical results show the proposed approach outperforms baselines.

Abstract

Human preference plays a crucial role in the refinement of large language models (LLMs). However, collecting human preference feedback is costly and most existing datasets neglect the correlation between personalization and preferences. To address this issue, we introduce Fair-PP, a synthetic dataset of personalized preferences targeting social equity, derived from real-world social survey data, which includes 28 social groups, 98 equity topics, and 5 personal preference dimensions. Leveraging GPT-4o-mini, we engage in role-playing based on seven representative persona portrayals guided by existing social survey data, yielding a total of 238,623 preference records. Through Fair-PP, we also contribute (i) An automated framework for generating preference data, along with a more fine-grained dataset of personalized preferences; (ii) analysis of the positioning of the existing mainstream LLMs across five major global regions within the personalized preference space; and (iii) a sample reweighting method for personalized preference alignment, enabling alignment with a target persona while maximizing the divergence from other personas. Empirical experiments show our method outperforms the baselines.